2023-01-11T21:14:24.9329291Z Requested labels: linux.8xlarge.nvidia.gpu
2023-01-11T21:14:24.9329377Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/trunk/91627
2023-01-11T21:14:24.9329478Z Reusable workflow chain:
2023-01-11T21:14:24.9329518Z pytorch/pytorch/.github/workflows/trunk.yml@refs/tags/ciflow/trunk/91627 (8419ddda87c8a47eacc63b54bc7ec98c1f27c26e)
2023-01-11T21:14:24.9329553Z -> pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/trunk/91627 (8419ddda87c8a47eacc63b54bc7ec98c1f27c26e)
2023-01-11T21:14:24.9329578Z Waiting for a runner to pick up this job...
2023-01-11T21:14:25.4573017Z Job is about to start running on the runner: i-0ac6ae9883f837720 (organization)
2023-01-11T21:14:30.3522844Z Current runner version: '2.300.2'
2023-01-11T21:14:30.3530521Z Runner name: 'i-0ac6ae9883f837720'
2023-01-11T21:14:30.3531235Z Runner group name: 'Default'
2023-01-11T21:14:30.3531959Z Machine name: 'ip-10-0-4-69'
2023-01-11T21:14:30.3534931Z ##[group]GITHUB_TOKEN Permissions
2023-01-11T21:14:30.3535812Z Actions: write
2023-01-11T21:14:30.3536266Z Checks: write
2023-01-11T21:14:30.3536759Z Contents: write
2023-01-11T21:14:30.3537143Z Deployments: write
2023-01-11T21:14:30.3537584Z Discussions: write
2023-01-11T21:14:30.3538055Z Issues: write
2023-01-11T21:14:30.3538420Z Metadata: read
2023-01-11T21:14:30.3538883Z Packages: write
2023-01-11T21:14:30.3539312Z Pages: write
2023-01-11T21:14:30.3539699Z PullRequests: write
2023-01-11T21:14:30.3540263Z RepositoryProjects: write
2023-01-11T21:14:30.3540776Z SecurityEvents: write
2023-01-11T21:14:30.3541170Z Statuses: write
2023-01-11T21:14:30.3541600Z ##[endgroup]
2023-01-11T21:14:30.3546061Z Secret source: Actions
2023-01-11T21:14:30.3546953Z Prepare workflow directory
2023-01-11T21:14:30.7066275Z Prepare all required actions
2023-01-11T21:14:30.7299969Z Getting action download info
2023-01-11T21:14:30.9427258Z Download action repository 'pytorch/test-infra@main' (SHA:2c225610d00fb13c04fcd60389d3e4d8326167c3)
2023-01-11T21:14:31.2762456Z Download action repository 'pytorch/pytorch@master' (SHA:c5836153f5332ca83d5cacde38f2829a4d54793e)
2023-01-11T21:14:35.0725628Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2023-01-11T21:14:35.3745303Z Getting action download info
2023-01-11T21:14:35.5450524Z Download action repository 'malfet/checkout@silent-checkout' (SHA:c7b8fef48edfe1bca0044a44b1f7f7c4318a3076)
2023-01-11T21:14:35.7479911Z Getting action download info
2023-01-11T21:14:35.9034988Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482)
2023-01-11T21:14:36.0471280Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml
2023-01-11T21:14:36.0474165Z ##[group] Inputs
2023-01-11T21:14:36.0474548Z   build-environment: linux-bionic-cuda11.7-py3.10-gcc7
2023-01-11T21:14:36.0475976Z   test-matrix: { include: [
  { config: "default", shard: 1, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 2, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 3, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 4, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "functorch", shard: 1, num_shards: 1, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "nogpu_AVX512", shard: 1, num_shards: 1, runner: "linux.2xlarge" },
  { config: "nogpu_NO_AVX2", shard: 1, num_shards: 1, runner: "linux.2xlarge" },
  { config: "jit_legacy", shard: 1, num_shards: 1, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "distributed", shard: 1, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "distributed", shard: 2, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "distributed", shard: 3, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
]}

2023-01-11T21:14:36.0477489Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T21:14:36.0477980Z   sync-tag: 
2023-01-11T21:14:36.0479009Z   timeout-minutes: 240
2023-01-11T21:14:36.0479275Z   use-gha: 
2023-01-11T21:14:36.0479543Z ##[endgroup]
2023-01-11T21:14:36.0480334Z Complete job name: linux-bionic-cuda11.7-py3.10-gcc7 / test (distributed, 2, 3, linux.8xlarge.nvidia.gpu)
2023-01-11T21:14:36.1544447Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main
2023-01-11T21:14:36.1544836Z with:
2023-01-11T21:14:36.1545404Z   github-secret: ***
2023-01-11T21:14:36.1545874Z   instructions: All testing is done inside the container, to start an interactive session run:
  docker exec -it $(docker container ps --format '{{.ID}}') bash

2023-01-11T21:14:36.1546326Z   activate-with-label: false
2023-01-11T21:14:36.1546755Z   label: with-ssh
2023-01-11T21:14:36.1546999Z   remove-existing-keys: true
2023-01-11T21:14:36.1547261Z env:
2023-01-11T21:14:36.1547499Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:14:36.1547746Z ##[endgroup]
2023-01-11T21:14:36.2958362Z ciflow reference detected, attempting to extract PR number
2023-01-11T21:14:36.7067704Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys
2023-01-11T21:14:36.7929407Z No SSH keys found for user pytorch-bot[bot]
2023-01-11T21:14:36.7929832Z Grabbing public ssh keys from https://github.com/LucaLumetti.keys
2023-01-11T21:14:36.8791796Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh
2023-01-11T21:14:36.8813354Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys
2023-01-11T21:14:36.8857301Z Login using: ssh ec2-user@ec2-54-166-140-4.compute-1.amazonaws.com
2023-01-11T21:14:36.8858174Z All testing is done inside the container, to start an interactive session run:
2023-01-11T21:14:36.8858678Z   docker exec -it $(docker container ps --format '{{.ID}}') bash
2023-01-11T21:14:36.9151462Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master
2023-01-11T21:14:36.9152203Z with:
2023-01-11T21:14:36.9152463Z   submodules: recursive
2023-01-11T21:14:36.9152720Z   fetch-depth: 0
2023-01-11T21:14:36.9152949Z env:
2023-01-11T21:14:36.9153166Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:14:36.9153422Z ##[endgroup]
2023-01-11T21:14:36.9450119Z ##[group]Run retry () {
2023-01-11T21:14:36.9450428Z retry () {
2023-01-11T21:14:36.9450731Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2023-01-11T21:14:36.9451020Z }
2023-01-11T21:14:36.9451257Z echo "${GITHUB_WORKSPACE}"
2023-01-11T21:14:36.9451547Z if [ -z "${NO_SUDO}" ]; then
2023-01-11T21:14:36.9451852Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2023-01-11T21:14:36.9452113Z else
2023-01-11T21:14:36.9452382Z   retry rm -rf "${GITHUB_WORKSPACE}"
2023-01-11T21:14:36.9452652Z fi
2023-01-11T21:14:36.9452925Z mkdir "${GITHUB_WORKSPACE}"
2023-01-11T21:14:36.9472223Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:14:36.9472544Z env:
2023-01-11T21:14:36.9472789Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:14:36.9473044Z   NO_SUDO: 
2023-01-11T21:14:36.9473263Z ##[endgroup]
2023-01-11T21:14:36.9607621Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T21:14:40.0394285Z ##[group]Run malfet/checkout@silent-checkout
2023-01-11T21:14:40.0394571Z with:
2023-01-11T21:14:40.0394841Z   ref: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:14:40.0395109Z   fetch-depth: 0
2023-01-11T21:14:40.0395352Z   submodules: recursive
2023-01-11T21:14:40.0395607Z   quiet-checkout: true
2023-01-11T21:14:40.0395859Z   repository: pytorch/pytorch
2023-01-11T21:14:40.0396289Z   token: ***
2023-01-11T21:14:40.0396525Z   ssh-strict: true
2023-01-11T21:14:40.0396788Z   persist-credentials: true
2023-01-11T21:14:40.0397042Z   clean: true
2023-01-11T21:14:40.0397272Z   lfs: false
2023-01-11T21:14:40.0397520Z   set-safe-directory: true
2023-01-11T21:14:40.0397745Z env:
2023-01-11T21:14:40.0397976Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:14:40.0398225Z ##[endgroup]
2023-01-11T21:14:40.1945864Z Syncing repository: pytorch/pytorch
2023-01-11T21:14:40.1947705Z ##[group]Getting Git version info
2023-01-11T21:14:40.1948246Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2023-01-11T21:14:40.1948826Z [command]/usr/bin/git version
2023-01-11T21:14:40.1949092Z git version 2.38.1
2023-01-11T21:14:40.1965768Z ##[endgroup]
2023-01-11T21:14:40.1986619Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/e2490b06-d4be-4697-8b62-8355f41531c6' before making global git config changes
2023-01-11T21:14:40.1987910Z Adding repository directory to the temporary git global config as a safe directory
2023-01-11T21:14:40.1994979Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T21:14:40.2043114Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2023-01-11T21:14:40.2050137Z ##[group]Initializing the repository
2023-01-11T21:14:40.2054082Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T21:14:40.2087492Z hint: Using 'master' as the name for the initial branch. This default branch name
2023-01-11T21:14:40.2087973Z hint: is subject to change. To configure the initial branch name to use in all
2023-01-11T21:14:40.2088662Z hint: of your new repositories, which will suppress this warning, call:
2023-01-11T21:14:40.2088959Z hint: 
2023-01-11T21:14:40.2089325Z hint: 	git config --global init.defaultBranch <name>
2023-01-11T21:14:40.2089772Z hint: 
2023-01-11T21:14:40.2090160Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2023-01-11T21:14:40.2090633Z hint: 'development'. The just-created branch can be renamed via this command:
2023-01-11T21:14:40.2090961Z hint: 
2023-01-11T21:14:40.2091389Z hint: 	git branch -m <name>
2023-01-11T21:14:40.2091889Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2023-01-11T21:14:40.2102219Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2023-01-11T21:14:40.2139014Z ##[endgroup]
2023-01-11T21:14:40.2139486Z ##[group]Disabling automatic garbage collection
2023-01-11T21:14:40.2143796Z [command]/usr/bin/git config --local gc.auto 0
2023-01-11T21:14:40.2176509Z ##[endgroup]
2023-01-11T21:14:40.2177498Z ##[group]Setting up auth
2023-01-11T21:14:40.2187020Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2023-01-11T21:14:40.2223593Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2023-01-11T21:14:40.2525355Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2023-01-11T21:14:40.2557319Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2023-01-11T21:14:40.2853333Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2023-01-11T21:14:40.2908619Z ##[endgroup]
2023-01-11T21:14:40.2909071Z ##[group]Fetching the repository
2023-01-11T21:14:40.2918333Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2023-01-11T21:15:37.2413308Z [command]/usr/bin/git rev-parse --verify --quiet 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e^{object}
2023-01-11T21:15:37.2444308Z 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:15:37.2452203Z ##[endgroup]
2023-01-11T21:15:37.2452702Z ##[group]Determining the checkout info
2023-01-11T21:15:37.2453169Z ##[endgroup]
2023-01-11T21:15:37.2453650Z ##[group]Checking out the ref
2023-01-11T21:15:37.2458600Z [command]/usr/bin/git checkout --quiet --force 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:15:38.9906662Z ##[endgroup]
2023-01-11T21:15:38.9907402Z ##[group]Setting up auth for fetching submodules
2023-01-11T21:15:38.9913740Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2023-01-11T21:15:38.9971471Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2023-01-11T21:15:39.0004809Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2023-01-11T21:15:39.0038586Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2023-01-11T21:15:39.0071249Z ##[endgroup]
2023-01-11T21:15:39.0072132Z ##[group]Fetching submodules
2023-01-11T21:15:39.0077719Z [command]/usr/bin/git submodule sync --recursive
2023-01-11T21:15:39.0403028Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2023-01-11T21:15:39.0714476Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2023-01-11T21:15:39.0717339Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2023-01-11T21:15:39.0720384Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2023-01-11T21:15:39.0723565Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2023-01-11T21:15:39.0726921Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK'
2023-01-11T21:15:39.0730613Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2023-01-11T21:15:39.0734226Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2023-01-11T21:15:39.0737865Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2023-01-11T21:15:39.0741754Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2023-01-11T21:15:39.0745716Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub'
2023-01-11T21:15:39.0750068Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2023-01-11T21:15:39.0755029Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2023-01-11T21:15:39.0759287Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2023-01-11T21:15:39.0764169Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2023-01-11T21:15:39.0768655Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2023-01-11T21:15:39.0772983Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2023-01-11T21:15:39.0778606Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2023-01-11T21:15:39.0783545Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2023-01-11T21:15:39.0788745Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2023-01-11T21:15:39.0795030Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2023-01-11T21:15:39.0800385Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2023-01-11T21:15:39.0806059Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake'
2023-01-11T21:15:39.0811825Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2023-01-11T21:15:39.0817639Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2023-01-11T21:15:39.0823677Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2023-01-11T21:15:39.0829832Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse'
2023-01-11T21:15:39.0836860Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2023-01-11T21:15:39.0843118Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2023-01-11T21:15:39.0849718Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt'
2023-01-11T21:15:39.0856307Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2023-01-11T21:15:39.0863202Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2023-01-11T21:15:39.0870061Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2023-01-11T21:15:39.0877882Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2023-01-11T21:15:39.0884940Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2023-01-11T21:15:39.0892399Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum'
2023-01-11T21:15:39.0899649Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2023-01-11T21:15:39.0907144Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six'
2023-01-11T21:15:39.0915520Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2023-01-11T21:15:39.0923299Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb'
2023-01-11T21:15:39.0931410Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2023-01-11T21:15:39.0939515Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd'
2023-01-11T21:15:39.0969661Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2023-01-11T21:15:39.3968943Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2023-01-11T21:15:39.6151807Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2023-01-11T21:15:39.8258481Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2023-01-11T21:15:40.1505183Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'...
2023-01-11T21:15:40.4501396Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2023-01-11T21:15:42.6475130Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2023-01-11T21:15:48.7730332Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2023-01-11T21:15:49.1984952Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2023-01-11T21:15:49.8002124Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'...
2023-01-11T21:15:51.3371797Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2023-01-11T21:15:52.7732280Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2023-01-11T21:15:54.1964102Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2023-01-11T21:16:01.2793392Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2023-01-11T21:16:02.1747997Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2023-01-11T21:16:04.0910541Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2023-01-11T21:16:05.2613424Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2023-01-11T21:16:05.5321375Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2023-01-11T21:16:06.0563487Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2023-01-11T21:16:06.4045056Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2023-01-11T21:16:08.2607119Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2023-01-11T21:16:08.7846861Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'...
2023-01-11T21:16:09.4055187Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2023-01-11T21:16:10.6144580Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2023-01-11T21:16:12.3114718Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2023-01-11T21:16:12.6947709Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'...
2023-01-11T21:16:13.1175879Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2023-01-11T21:16:19.3263693Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2023-01-11T21:16:21.1093793Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'...
2023-01-11T21:16:21.5879848Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2023-01-11T21:16:21.8154254Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2023-01-11T21:16:28.2845861Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2023-01-11T21:16:28.5616365Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2023-01-11T21:16:28.8157427Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2023-01-11T21:16:29.7716886Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'...
2023-01-11T21:16:30.0196361Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2023-01-11T21:16:30.3627579Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'...
2023-01-11T21:16:30.6878365Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2023-01-11T21:16:31.2689372Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'...
2023-01-11T21:16:33.5428160Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2023-01-11T21:16:34.1676756Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'...
2023-01-11T21:16:36.7205176Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2023-01-11T21:16:36.7329968Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2023-01-11T21:16:36.7426147Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2023-01-11T21:16:36.7715583Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2023-01-11T21:16:36.7990679Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c'
2023-01-11T21:16:36.8412278Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2023-01-11T21:16:37.6019329Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba'
2023-01-11T21:16:37.6270918Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2023-01-11T21:16:37.7490852Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c'
2023-01-11T21:16:37.7908880Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4'
2023-01-11T21:16:38.1499581Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3'
2023-01-11T21:16:38.6680263Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a'
2023-01-11T21:16:38.9702315Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2023-01-11T21:16:39.0260956Z Submodule path 'third_party/fbgemm': checked out '80d64206c07879fd4683be66873de7cefa1a0a71'
2023-01-11T21:16:39.0280505Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2023-01-11T21:16:39.0284701Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T21:16:39.0289576Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2023-01-11T21:16:39.0294487Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T21:16:39.0321189Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2023-01-11T21:16:40.0171297Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2023-01-11T21:16:40.6094974Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2023-01-11T21:16:41.6582691Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2023-01-11T21:16:41.9945569Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2023-01-11T21:16:42.1179544Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2023-01-11T21:16:42.1892637Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2023-01-11T21:16:42.2006530Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5'
2023-01-11T21:16:42.3077861Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa'
2023-01-11T21:16:42.3521056Z Submodule path 'third_party/fmt': checked out '7bdf0628b1276379886c7f6dda2cef2b3b374f0b'
2023-01-11T21:16:42.3621937Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2023-01-11T21:16:42.4105907Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2023-01-11T21:16:42.4393163Z Submodule path 'third_party/gloo': checked out '4a5e339b764261d20fc409071dc7a8b8989aa195'
2023-01-11T21:16:42.4942122Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2023-01-11T21:16:42.5076618Z Submodule path 'third_party/ideep': checked out 'e533c771a1e75a1c225c14b2261eefa62681d9e6'
2023-01-11T21:16:42.5093512Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2023-01-11T21:16:42.5120606Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2023-01-11T21:16:51.5131713Z Submodule path 'third_party/ideep/mkl-dnn': checked out '404ad76ee633c939d705eb583ffe50a806969d5e'
2023-01-11T21:16:51.5152694Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T21:16:51.5180944Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'...
2023-01-11T21:17:00.6869622Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out 'fbec3e25a559ee252022ae066817b204e106a6ba'
2023-01-11T21:17:00.6987665Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724'
2023-01-11T21:17:00.7159610Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2023-01-11T21:17:00.8269241Z Submodule path 'third_party/kineto': checked out '6c1629809068efd78a8d56b4aa479c7ec49ae562'
2023-01-11T21:17:00.8287939Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T21:17:00.8291315Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T21:17:00.8318425Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2023-01-11T21:17:02.0349432Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2023-01-11T21:17:03.1354805Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd'
2023-01-11T21:17:03.2005742Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2023-01-11T21:17:03.2255439Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52'
2023-01-11T21:17:03.2416473Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a'
2023-01-11T21:17:03.3753837Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2023-01-11T21:17:03.6915817Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b'
2023-01-11T21:17:03.6948565Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2023-01-11T21:17:03.6951862Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2023-01-11T21:17:03.6980495Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2023-01-11T21:17:05.2724223Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2023-01-11T21:17:06.2867259Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2023-01-11T21:17:06.3244910Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8'
2023-01-11T21:17:06.3421478Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f'
2023-01-11T21:17:06.3449108Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T21:17:06.3476857Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'...
2023-01-11T21:17:08.7139999Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8'
2023-01-11T21:17:08.7162324Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T21:17:08.7165323Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T21:17:08.7193564Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'...
2023-01-11T21:17:09.1881969Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'...
2023-01-11T21:17:10.1762857Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2023-01-11T21:17:10.2567703Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c'
2023-01-11T21:17:10.2584565Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T21:17:10.2612237Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'...
2023-01-11T21:17:10.5070342Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2023-01-11T21:17:10.5175148Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a'
2023-01-11T21:17:10.8421285Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2023-01-11T21:17:10.8444531Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2023-01-11T21:17:10.8448060Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2023-01-11T21:17:10.8476239Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2023-01-11T21:17:11.2967025Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2023-01-11T21:17:12.3079650Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2023-01-11T21:17:12.3901926Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2023-01-11T21:17:12.3994629Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2023-01-11T21:17:12.4119950Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413'
2023-01-11T21:17:12.4507211Z Submodule path 'third_party/pybind11': checked out '80dc998efced8ceb2be59756668a7e90e8bef917'
2023-01-11T21:17:12.4604813Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7'
2023-01-11T21:17:12.4938137Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2023-01-11T21:17:12.5044817Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a'
2023-01-11T21:17:12.5584221Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff'
2023-01-11T21:17:12.6961526Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9'
2023-01-11T21:17:12.7261275Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2023-01-11T21:17:12.7280036Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2023-01-11T21:17:12.7283350Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2023-01-11T21:17:12.7286687Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2023-01-11T21:17:12.7290175Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T21:17:12.7318142Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2023-01-11T21:17:13.9717863Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2023-01-11T21:17:14.2443106Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2023-01-11T21:17:15.6373052Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2023-01-11T21:17:17.2975819Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2023-01-11T21:17:17.3166042Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2023-01-11T21:17:17.3929692Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2023-01-11T21:17:17.4258411Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2023-01-11T21:17:17.4277137Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T21:17:17.4305233Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2023-01-11T21:17:17.6676253Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2023-01-11T21:17:17.8330521Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8'
2023-01-11T21:17:17.8363666Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2023-01-11T21:17:17.8703768Z Entering 'android/libs/fbjni'
2023-01-11T21:17:17.8750212Z Entering 'third_party/FP16'
2023-01-11T21:17:17.8795902Z Entering 'third_party/FXdiv'
2023-01-11T21:17:17.8840227Z Entering 'third_party/NNPACK'
2023-01-11T21:17:17.8885959Z Entering 'third_party/QNNPACK'
2023-01-11T21:17:17.8930439Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T21:17:17.8975019Z Entering 'third_party/XNNPACK'
2023-01-11T21:17:17.9029989Z Entering 'third_party/benchmark'
2023-01-11T21:17:17.9075226Z Entering 'third_party/cpuinfo'
2023-01-11T21:17:17.9120021Z Entering 'third_party/cub'
2023-01-11T21:17:17.9164125Z Entering 'third_party/cudnn_frontend'
2023-01-11T21:17:17.9213627Z Entering 'third_party/cutlass'
2023-01-11T21:17:17.9265714Z Entering 'third_party/eigen'
2023-01-11T21:17:17.9311616Z Entering 'third_party/fbgemm'
2023-01-11T21:17:17.9355674Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T21:17:17.9398386Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T21:17:17.9442557Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T21:17:17.9485392Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T21:17:17.9529590Z Entering 'third_party/flatbuffers'
2023-01-11T21:17:17.9575728Z Entering 'third_party/fmt'
2023-01-11T21:17:17.9619453Z Entering 'third_party/foxi'
2023-01-11T21:17:17.9662999Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T21:17:17.9709178Z Entering 'third_party/gloo'
2023-01-11T21:17:17.9755996Z Entering 'third_party/googletest'
2023-01-11T21:17:17.9799438Z Entering 'third_party/ideep'
2023-01-11T21:17:17.9841733Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T21:17:17.9887097Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T21:17:17.9938330Z Entering 'third_party/ios-cmake'
2023-01-11T21:17:17.9985481Z Entering 'third_party/ittapi'
2023-01-11T21:17:18.0028741Z Entering 'third_party/kineto'
2023-01-11T21:17:18.0072650Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T21:17:18.0119155Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T21:17:18.0163662Z Entering 'third_party/nccl/nccl'
2023-01-11T21:17:18.0207190Z Entering 'third_party/neon2sse'
2023-01-11T21:17:18.0250065Z Entering 'third_party/nlohmann'
2023-01-11T21:17:18.0295719Z Entering 'third_party/onnx'
2023-01-11T21:17:18.0354127Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T21:17:18.0398725Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T21:17:18.0445000Z Entering 'third_party/onnx-tensorrt'
2023-01-11T21:17:18.0488718Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T21:17:18.0537759Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T21:17:18.0582731Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T21:17:18.0626579Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T21:17:18.0677423Z Entering 'third_party/pocketfft'
2023-01-11T21:17:18.0722797Z Entering 'third_party/protobuf'
2023-01-11T21:17:18.0770525Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T21:17:18.0813198Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T21:17:18.0858250Z Entering 'third_party/psimd'
2023-01-11T21:17:18.0902225Z Entering 'third_party/pthreadpool'
2023-01-11T21:17:18.0946075Z Entering 'third_party/pybind11'
2023-01-11T21:17:18.0988867Z Entering 'third_party/python-enum'
2023-01-11T21:17:18.1031473Z Entering 'third_party/python-peachpy'
2023-01-11T21:17:18.1074306Z Entering 'third_party/python-six'
2023-01-11T21:17:18.1116715Z Entering 'third_party/sleef'
2023-01-11T21:17:18.1160129Z Entering 'third_party/tbb'
2023-01-11T21:17:18.1206419Z Entering 'third_party/tensorpipe'
2023-01-11T21:17:18.1249665Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T21:17:18.1291258Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T21:17:18.1334669Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T21:17:18.1376541Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T21:17:18.1418962Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T21:17:18.1463939Z Entering 'third_party/zstd'
2023-01-11T21:17:18.1518457Z ##[endgroup]
2023-01-11T21:17:18.1518975Z ##[group]Persisting credentials for submodules
2023-01-11T21:17:18.1526369Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :
2023-01-11T21:17:18.1842651Z Entering 'android/libs/fbjni'
2023-01-11T21:17:18.1885333Z Entering 'third_party/FP16'
2023-01-11T21:17:18.1927388Z Entering 'third_party/FXdiv'
2023-01-11T21:17:18.1969710Z Entering 'third_party/NNPACK'
2023-01-11T21:17:18.2012881Z Entering 'third_party/QNNPACK'
2023-01-11T21:17:18.2055715Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T21:17:18.2099361Z Entering 'third_party/XNNPACK'
2023-01-11T21:17:18.2153711Z Entering 'third_party/benchmark'
2023-01-11T21:17:18.2196934Z Entering 'third_party/cpuinfo'
2023-01-11T21:17:18.2239959Z Entering 'third_party/cub'
2023-01-11T21:17:18.2283714Z Entering 'third_party/cudnn_frontend'
2023-01-11T21:17:18.2331807Z Entering 'third_party/cutlass'
2023-01-11T21:17:18.2381572Z Entering 'third_party/eigen'
2023-01-11T21:17:18.2427289Z Entering 'third_party/fbgemm'
2023-01-11T21:17:18.2469894Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T21:17:18.2512362Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T21:17:18.2554799Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T21:17:18.2597807Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T21:17:18.2641260Z Entering 'third_party/flatbuffers'
2023-01-11T21:17:18.2686722Z Entering 'third_party/fmt'
2023-01-11T21:17:18.2729918Z Entering 'third_party/foxi'
2023-01-11T21:17:18.2772856Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T21:17:18.2816243Z Entering 'third_party/gloo'
2023-01-11T21:17:18.2859886Z Entering 'third_party/googletest'
2023-01-11T21:17:18.2903363Z Entering 'third_party/ideep'
2023-01-11T21:17:18.2945728Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T21:17:18.2990552Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T21:17:18.3041574Z Entering 'third_party/ios-cmake'
2023-01-11T21:17:18.3084516Z Entering 'third_party/ittapi'
2023-01-11T21:17:18.3127190Z Entering 'third_party/kineto'
2023-01-11T21:17:18.3170710Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T21:17:18.3214754Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T21:17:18.3260215Z Entering 'third_party/nccl/nccl'
2023-01-11T21:17:18.3302980Z Entering 'third_party/neon2sse'
2023-01-11T21:17:18.3346418Z Entering 'third_party/nlohmann'
2023-01-11T21:17:18.3391249Z Entering 'third_party/onnx'
2023-01-11T21:17:18.3448425Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T21:17:18.3491877Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T21:17:18.3536705Z Entering 'third_party/onnx-tensorrt'
2023-01-11T21:17:18.3579564Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T21:17:18.3626733Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T21:17:18.3669998Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T21:17:18.3714061Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T21:17:18.3762633Z Entering 'third_party/pocketfft'
2023-01-11T21:17:18.3805909Z Entering 'third_party/protobuf'
2023-01-11T21:17:18.3852030Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T21:17:18.3894904Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T21:17:18.3939770Z Entering 'third_party/psimd'
2023-01-11T21:17:18.3982703Z Entering 'third_party/pthreadpool'
2023-01-11T21:17:18.4025547Z Entering 'third_party/pybind11'
2023-01-11T21:17:18.4068490Z Entering 'third_party/python-enum'
2023-01-11T21:17:18.4111166Z Entering 'third_party/python-peachpy'
2023-01-11T21:17:18.4154023Z Entering 'third_party/python-six'
2023-01-11T21:17:18.4197057Z Entering 'third_party/sleef'
2023-01-11T21:17:18.4240608Z Entering 'third_party/tbb'
2023-01-11T21:17:18.4285864Z Entering 'third_party/tensorpipe'
2023-01-11T21:17:18.4328709Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T21:17:18.4370933Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T21:17:18.4413452Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T21:17:18.4455802Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T21:17:18.4497432Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T21:17:18.4543119Z Entering 'third_party/zstd'
2023-01-11T21:17:18.4599851Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url
2023-01-11T21:17:18.4921709Z Entering 'android/libs/fbjni'
2023-01-11T21:17:18.4962201Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2023-01-11T21:17:18.4979622Z Entering 'third_party/FP16'
2023-01-11T21:17:18.5019945Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2023-01-11T21:17:18.5038268Z Entering 'third_party/FXdiv'
2023-01-11T21:17:18.5079048Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2023-01-11T21:17:18.5096103Z Entering 'third_party/NNPACK'
2023-01-11T21:17:18.5136717Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2023-01-11T21:17:18.5155232Z Entering 'third_party/QNNPACK'
2023-01-11T21:17:18.5194999Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config	remote.origin.url
2023-01-11T21:17:18.5212681Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T21:17:18.5253295Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2023-01-11T21:17:18.5270909Z Entering 'third_party/XNNPACK'
2023-01-11T21:17:18.5311269Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2023-01-11T21:17:18.5340290Z Entering 'third_party/benchmark'
2023-01-11T21:17:18.5380130Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2023-01-11T21:17:18.5398645Z Entering 'third_party/cpuinfo'
2023-01-11T21:17:18.5438430Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2023-01-11T21:17:18.5456421Z Entering 'third_party/cub'
2023-01-11T21:17:18.5496916Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config	remote.origin.url
2023-01-11T21:17:18.5515200Z Entering 'third_party/cudnn_frontend'
2023-01-11T21:17:18.5555298Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2023-01-11T21:17:18.5578029Z Entering 'third_party/cutlass'
2023-01-11T21:17:18.5617251Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2023-01-11T21:17:18.5643398Z Entering 'third_party/eigen'
2023-01-11T21:17:18.5683342Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2023-01-11T21:17:18.5703223Z Entering 'third_party/fbgemm'
2023-01-11T21:17:18.5744116Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2023-01-11T21:17:18.5762640Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T21:17:18.5801736Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2023-01-11T21:17:18.5818867Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T21:17:18.5859507Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2023-01-11T21:17:18.5878653Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T21:17:18.5918238Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2023-01-11T21:17:18.5935471Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T21:17:18.5975529Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2023-01-11T21:17:18.5994407Z Entering 'third_party/flatbuffers'
2023-01-11T21:17:18.6034442Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2023-01-11T21:17:18.6053787Z Entering 'third_party/fmt'
2023-01-11T21:17:18.6093664Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2023-01-11T21:17:18.6111209Z Entering 'third_party/foxi'
2023-01-11T21:17:18.6151722Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2023-01-11T21:17:18.6170087Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T21:17:18.6210549Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2023-01-11T21:17:18.6228175Z Entering 'third_party/gloo'
2023-01-11T21:17:18.6268533Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2023-01-11T21:17:18.6287333Z Entering 'third_party/googletest'
2023-01-11T21:17:18.6327114Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2023-01-11T21:17:18.6345877Z Entering 'third_party/ideep'
2023-01-11T21:17:18.6385727Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2023-01-11T21:17:18.6403071Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T21:17:18.6442715Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2023-01-11T21:17:18.6463277Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T21:17:18.6503946Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config	remote.origin.url
2023-01-11T21:17:18.6529903Z Entering 'third_party/ios-cmake'
2023-01-11T21:17:18.6569473Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config	remote.origin.url
2023-01-11T21:17:18.6587022Z Entering 'third_party/ittapi'
2023-01-11T21:17:18.6627685Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2023-01-11T21:17:18.6645889Z Entering 'third_party/kineto'
2023-01-11T21:17:18.6685498Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2023-01-11T21:17:18.6703012Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T21:17:18.6742932Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2023-01-11T21:17:18.6761084Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T21:17:18.6800762Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2023-01-11T21:17:18.6819786Z Entering 'third_party/nccl/nccl'
2023-01-11T21:17:18.6859395Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2023-01-11T21:17:18.6877982Z Entering 'third_party/neon2sse'
2023-01-11T21:17:18.6917743Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config	remote.origin.url
2023-01-11T21:17:18.6935085Z Entering 'third_party/nlohmann'
2023-01-11T21:17:18.6975147Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2023-01-11T21:17:18.6994917Z Entering 'third_party/onnx'
2023-01-11T21:17:18.7034499Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2023-01-11T21:17:18.7064772Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T21:17:18.7104909Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2023-01-11T21:17:18.7123371Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T21:17:18.7163284Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2023-01-11T21:17:18.7182883Z Entering 'third_party/onnx-tensorrt'
2023-01-11T21:17:18.7223060Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config	remote.origin.url
2023-01-11T21:17:18.7241028Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T21:17:18.7281714Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config	remote.origin.url
2023-01-11T21:17:18.7304826Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T21:17:18.7345897Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2023-01-11T21:17:18.7365297Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T21:17:18.7404969Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2023-01-11T21:17:18.7422453Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T21:17:18.7463674Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2023-01-11T21:17:18.7487025Z Entering 'third_party/pocketfft'
2023-01-11T21:17:18.7526772Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2023-01-11T21:17:18.7544055Z Entering 'third_party/protobuf'
2023-01-11T21:17:18.7584125Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2023-01-11T21:17:18.7606200Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T21:17:18.7645847Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2023-01-11T21:17:18.7663068Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T21:17:18.7703371Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2023-01-11T21:17:18.7723373Z Entering 'third_party/psimd'
2023-01-11T21:17:18.7763409Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2023-01-11T21:17:18.7780793Z Entering 'third_party/pthreadpool'
2023-01-11T21:17:18.7820622Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2023-01-11T21:17:18.7838790Z Entering 'third_party/pybind11'
2023-01-11T21:17:18.7879281Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2023-01-11T21:17:18.7897822Z Entering 'third_party/python-enum'
2023-01-11T21:17:18.7938325Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config	remote.origin.url
2023-01-11T21:17:18.7956630Z Entering 'third_party/python-peachpy'
2023-01-11T21:17:18.7997345Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2023-01-11T21:17:18.8014983Z Entering 'third_party/python-six'
2023-01-11T21:17:18.8054476Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config	remote.origin.url
2023-01-11T21:17:18.8071790Z Entering 'third_party/sleef'
2023-01-11T21:17:18.8111636Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2023-01-11T21:17:18.8129646Z Entering 'third_party/tbb'
2023-01-11T21:17:18.8169505Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config	remote.origin.url
2023-01-11T21:17:18.8189638Z Entering 'third_party/tensorpipe'
2023-01-11T21:17:18.8230056Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2023-01-11T21:17:18.8247679Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T21:17:18.8286821Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2023-01-11T21:17:18.8304174Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T21:17:18.8344452Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2023-01-11T21:17:18.8362696Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T21:17:18.8402263Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2023-01-11T21:17:18.8420224Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T21:17:18.8459683Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2023-01-11T21:17:18.8477208Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T21:17:18.8517534Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2023-01-11T21:17:18.8537707Z Entering 'third_party/zstd'
2023-01-11T21:17:18.8578453Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config	remote.origin.url
2023-01-11T21:17:18.9477058Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2023-01-11T21:17:18.9800140Z Entering 'android/libs/fbjni'
2023-01-11T21:17:18.9843913Z Entering 'third_party/FP16'
2023-01-11T21:17:18.9887836Z Entering 'third_party/FXdiv'
2023-01-11T21:17:18.9931463Z Entering 'third_party/NNPACK'
2023-01-11T21:17:18.9976320Z Entering 'third_party/QNNPACK'
2023-01-11T21:17:19.0021501Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T21:17:19.0065819Z Entering 'third_party/XNNPACK'
2023-01-11T21:17:19.0121944Z Entering 'third_party/benchmark'
2023-01-11T21:17:19.0165354Z Entering 'third_party/cpuinfo'
2023-01-11T21:17:19.0210124Z Entering 'third_party/cub'
2023-01-11T21:17:19.0253676Z Entering 'third_party/cudnn_frontend'
2023-01-11T21:17:19.0304249Z Entering 'third_party/cutlass'
2023-01-11T21:17:19.0356296Z Entering 'third_party/eigen'
2023-01-11T21:17:19.0402820Z Entering 'third_party/fbgemm'
2023-01-11T21:17:19.0446508Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T21:17:19.0489906Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T21:17:19.0534118Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T21:17:19.0577828Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T21:17:19.0622254Z Entering 'third_party/flatbuffers'
2023-01-11T21:17:19.0668700Z Entering 'third_party/fmt'
2023-01-11T21:17:19.0713301Z Entering 'third_party/foxi'
2023-01-11T21:17:19.0756841Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T21:17:19.0801485Z Entering 'third_party/gloo'
2023-01-11T21:17:19.0846126Z Entering 'third_party/googletest'
2023-01-11T21:17:19.0890419Z Entering 'third_party/ideep'
2023-01-11T21:17:19.0933936Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T21:17:19.0980335Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T21:17:19.1030942Z Entering 'third_party/ios-cmake'
2023-01-11T21:17:19.1075771Z Entering 'third_party/ittapi'
2023-01-11T21:17:19.1120108Z Entering 'third_party/kineto'
2023-01-11T21:17:19.1165327Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T21:17:19.1209083Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T21:17:19.1253921Z Entering 'third_party/nccl/nccl'
2023-01-11T21:17:19.1298860Z Entering 'third_party/neon2sse'
2023-01-11T21:17:19.1342787Z Entering 'third_party/nlohmann'
2023-01-11T21:17:19.1388817Z Entering 'third_party/onnx'
2023-01-11T21:17:19.1446174Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T21:17:19.1490025Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T21:17:19.1536041Z Entering 'third_party/onnx-tensorrt'
2023-01-11T21:17:19.1579098Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T21:17:19.1628347Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T21:17:19.1673553Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T21:17:19.1717924Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T21:17:19.1766854Z Entering 'third_party/pocketfft'
2023-01-11T21:17:19.1811160Z Entering 'third_party/protobuf'
2023-01-11T21:17:19.1859777Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T21:17:19.1903171Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T21:17:19.1948359Z Entering 'third_party/psimd'
2023-01-11T21:17:19.1993301Z Entering 'third_party/pthreadpool'
2023-01-11T21:17:19.2037079Z Entering 'third_party/pybind11'
2023-01-11T21:17:19.2080867Z Entering 'third_party/python-enum'
2023-01-11T21:17:19.2124116Z Entering 'third_party/python-peachpy'
2023-01-11T21:17:19.2168035Z Entering 'third_party/python-six'
2023-01-11T21:17:19.2212654Z Entering 'third_party/sleef'
2023-01-11T21:17:19.2256790Z Entering 'third_party/tbb'
2023-01-11T21:17:19.2303435Z Entering 'third_party/tensorpipe'
2023-01-11T21:17:19.2347292Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T21:17:19.2392514Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T21:17:19.2435793Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T21:17:19.2479255Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T21:17:19.2521814Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T21:17:19.2568290Z Entering 'third_party/zstd'
2023-01-11T21:17:19.2625905Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2023-01-11T21:17:19.2948353Z Entering 'android/libs/fbjni'
2023-01-11T21:17:19.2993021Z Entering 'third_party/FP16'
2023-01-11T21:17:19.3037341Z Entering 'third_party/FXdiv'
2023-01-11T21:17:19.3082055Z Entering 'third_party/NNPACK'
2023-01-11T21:17:19.3126072Z Entering 'third_party/QNNPACK'
2023-01-11T21:17:19.3170713Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T21:17:19.3215961Z Entering 'third_party/XNNPACK'
2023-01-11T21:17:19.3271292Z Entering 'third_party/benchmark'
2023-01-11T21:17:19.3315338Z Entering 'third_party/cpuinfo'
2023-01-11T21:17:19.3360194Z Entering 'third_party/cub'
2023-01-11T21:17:19.3403960Z Entering 'third_party/cudnn_frontend'
2023-01-11T21:17:19.3454206Z Entering 'third_party/cutlass'
2023-01-11T21:17:19.3504719Z Entering 'third_party/eigen'
2023-01-11T21:17:19.3551654Z Entering 'third_party/fbgemm'
2023-01-11T21:17:19.3596622Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T21:17:19.3640063Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T21:17:19.3683366Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T21:17:19.3727386Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T21:17:19.3771212Z Entering 'third_party/flatbuffers'
2023-01-11T21:17:19.3817013Z Entering 'third_party/fmt'
2023-01-11T21:17:19.3861598Z Entering 'third_party/foxi'
2023-01-11T21:17:19.3904862Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T21:17:19.3949594Z Entering 'third_party/gloo'
2023-01-11T21:17:19.3993407Z Entering 'third_party/googletest'
2023-01-11T21:17:19.4038386Z Entering 'third_party/ideep'
2023-01-11T21:17:19.4081621Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T21:17:19.4127602Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T21:17:19.4178732Z Entering 'third_party/ios-cmake'
2023-01-11T21:17:19.4223203Z Entering 'third_party/ittapi'
2023-01-11T21:17:19.4267616Z Entering 'third_party/kineto'
2023-01-11T21:17:19.4311699Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T21:17:19.4356384Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T21:17:19.4401582Z Entering 'third_party/nccl/nccl'
2023-01-11T21:17:19.4446153Z Entering 'third_party/neon2sse'
2023-01-11T21:17:19.4493194Z Entering 'third_party/nlohmann'
2023-01-11T21:17:19.4539396Z Entering 'third_party/onnx'
2023-01-11T21:17:19.4600144Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T21:17:19.4645751Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T21:17:19.4693735Z Entering 'third_party/onnx-tensorrt'
2023-01-11T21:17:19.4738557Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T21:17:19.4787261Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T21:17:19.4831417Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T21:17:19.4913136Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T21:17:19.4961879Z Entering 'third_party/pocketfft'
2023-01-11T21:17:19.5006277Z Entering 'third_party/protobuf'
2023-01-11T21:17:19.5054531Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T21:17:19.5099117Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T21:17:19.5144839Z Entering 'third_party/psimd'
2023-01-11T21:17:19.5188197Z Entering 'third_party/pthreadpool'
2023-01-11T21:17:19.5233311Z Entering 'third_party/pybind11'
2023-01-11T21:17:19.5279762Z Entering 'third_party/python-enum'
2023-01-11T21:17:19.5324139Z Entering 'third_party/python-peachpy'
2023-01-11T21:17:19.5367643Z Entering 'third_party/python-six'
2023-01-11T21:17:19.5412023Z Entering 'third_party/sleef'
2023-01-11T21:17:19.5455582Z Entering 'third_party/tbb'
2023-01-11T21:17:19.5500900Z Entering 'third_party/tensorpipe'
2023-01-11T21:17:19.5544683Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T21:17:19.5589136Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T21:17:19.5631155Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T21:17:19.5674679Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T21:17:19.5716690Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T21:17:19.5762502Z Entering 'third_party/zstd'
2023-01-11T21:17:19.5825479Z ##[endgroup]
2023-01-11T21:17:19.5870201Z [command]/usr/bin/git log -1 --format='%H'
2023-01-11T21:17:19.5899692Z '8419ddda87c8a47eacc63b54bc7ec98c1f27c26e'
2023-01-11T21:17:19.6079841Z Prepare all required actions
2023-01-11T21:17:19.6110132Z ##[group]Run ./.github/actions/setup-linux
2023-01-11T21:17:19.6110382Z env:
2023-01-11T21:17:19.6110615Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:17:19.6110863Z ##[endgroup]
2023-01-11T21:17:19.6129324Z ##[group]Run set -euo pipefail
2023-01-11T21:17:19.6129606Z set -euo pipefail
2023-01-11T21:17:19.6129861Z function get_ec2_metadata() {
2023-01-11T21:17:19.6130176Z   # Pulled from instance metadata endpoint for EC2
2023-01-11T21:17:19.6130627Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2023-01-11T21:17:19.6130994Z   category=$1
2023-01-11T21:17:19.6131298Z   curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2023-01-11T21:17:19.6131582Z }
2023-01-11T21:17:19.6131835Z echo "ami-id: $(get_ec2_metadata ami-id)"
2023-01-11T21:17:19.6132200Z echo "instance-id: $(get_ec2_metadata instance-id)"
2023-01-11T21:17:19.6132564Z echo "instance-type: $(get_ec2_metadata instance-type)"
2023-01-11T21:17:19.6132881Z echo "system info $(uname -a)"
2023-01-11T21:17:19.6146481Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:17:19.6146753Z env:
2023-01-11T21:17:19.6146964Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:17:19.6147197Z ##[endgroup]
2023-01-11T21:17:19.6248164Z ami-id: ami-096198a0bccc6bad4
2023-01-11T21:17:19.6310897Z instance-id: i-0ac6ae9883f837720
2023-01-11T21:17:19.6370041Z instance-type: g3.8xlarge
2023-01-11T21:17:19.6378752Z system info Linux ip-10-0-4-69.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
2023-01-11T21:17:19.6396678Z ##[group]Run if systemctl is-active --quiet docker; then
2023-01-11T21:17:19.6397032Z if systemctl is-active --quiet docker; then
2023-01-11T21:17:19.6397356Z     echo "Docker daemon is running...";
2023-01-11T21:17:19.6397638Z else
2023-01-11T21:17:19.6397936Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2023-01-11T21:17:19.6398239Z fi
2023-01-11T21:17:19.6410212Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:17:19.6410485Z env:
2023-01-11T21:17:19.6410719Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:17:19.6410970Z ##[endgroup]
2023-01-11T21:17:19.6461094Z Docker daemon is running...
2023-01-11T21:17:19.6479417Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2023-01-11T21:17:19.6479887Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2023-01-11T21:17:19.6480251Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2023-01-11T21:17:19.6480737Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \
2023-01-11T21:17:19.6481201Z     --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
2023-01-11T21:17:19.6493151Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:17:19.6493432Z env:
2023-01-11T21:17:19.6493670Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:17:19.6493934Z   AWS_RETRY_MODE: standard
2023-01-11T21:17:19.6494174Z   AWS_MAX_ATTEMPTS: 5
2023-01-11T21:17:19.6494436Z   AWS_DEFAULT_REGION: us-east-1
2023-01-11T21:17:19.6494693Z ##[endgroup]
2023-01-11T21:17:20.5906761Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2023-01-11T21:17:20.5907238Z Configure a credential helper to remove this warning. See
2023-01-11T21:17:20.5907746Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2023-01-11T21:17:20.5908013Z 
2023-01-11T21:17:20.5908419Z Login Succeeded
2023-01-11T21:17:20.5943099Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2023-01-11T21:17:20.5943486Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2023-01-11T21:17:20.5943967Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2023-01-11T21:17:20.5957800Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:17:20.5958075Z env:
2023-01-11T21:17:20.5958314Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:17:20.5958568Z ##[endgroup]
2023-01-11T21:17:20.6048466Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2023-01-11T21:17:20.6048838Z with:
2023-01-11T21:17:20.6049325Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T21:17:20.6049774Z env:
2023-01-11T21:17:20.6050007Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:17:20.6050259Z ##[endgroup]
2023-01-11T21:17:20.6065611Z ##[group]Run retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2023-01-11T21:17:20.6065959Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2023-01-11T21:17:20.6066312Z # ignore output since only exit code is used for conditional
2023-01-11T21:17:20.6066693Z # only pull docker image if it's not available locally
2023-01-11T21:17:20.6067090Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2023-01-11T21:17:20.6067478Z   retry docker pull "${DOCKER_IMAGE}"
2023-01-11T21:17:20.6067744Z fi
2023-01-11T21:17:20.6080529Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:17:20.6080823Z env:
2023-01-11T21:17:20.6081058Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:17:20.6081559Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T21:17:20.6082041Z ##[endgroup]
2023-01-11T21:17:20.8468768Z fd224c2e6c79d7fdec6408da598bf52bc5b201dd: Pulling from pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7
2023-01-11T21:17:20.8469523Z fb668870d8a7: Pulling fs layer
2023-01-11T21:17:20.8469805Z 4542784317be: Pulling fs layer
2023-01-11T21:17:20.8470096Z e0bec5df5af5: Pulling fs layer
2023-01-11T21:17:20.8470417Z 4053f75740ab: Pulling fs layer
2023-01-11T21:17:20.8470905Z 57e09105cdfd: Pulling fs layer
2023-01-11T21:17:20.8471177Z 606761d225e5: Pulling fs layer
2023-01-11T21:17:20.8471448Z 69473a703fb4: Pulling fs layer
2023-01-11T21:17:20.8471716Z a08ab4e0594b: Pulling fs layer
2023-01-11T21:17:20.8472297Z 4cd507bccac2: Pulling fs layer
2023-01-11T21:17:20.8472557Z fa92f16621a4: Pulling fs layer
2023-01-11T21:17:20.8472825Z 6dc2b05bd224: Pulling fs layer
2023-01-11T21:17:20.8473091Z ce4a87d45645: Pulling fs layer
2023-01-11T21:17:20.8473339Z 41860ea59b6c: Pulling fs layer
2023-01-11T21:17:20.8473699Z 87d0ffa55850: Pulling fs layer
2023-01-11T21:17:20.8473975Z f9f75aaba8d7: Pulling fs layer
2023-01-11T21:17:20.8474254Z 0c06be5c20e0: Pulling fs layer
2023-01-11T21:17:20.8474505Z d23c0a07b67c: Pulling fs layer
2023-01-11T21:17:20.8474769Z 1001f0d2f3d0: Pulling fs layer
2023-01-11T21:17:20.8475034Z e1c655e7ec0e: Pulling fs layer
2023-01-11T21:17:20.8475776Z a11b4b5fd784: Pulling fs layer
2023-01-11T21:17:20.8476255Z 69473a703fb4: Waiting
2023-01-11T21:17:20.8476758Z bc41eab7f454: Pulling fs layer
2023-01-11T21:17:20.8477252Z b8f759fd0191: Pulling fs layer
2023-01-11T21:17:20.8477724Z 41860ea59b6c: Waiting
2023-01-11T21:17:20.8482917Z f410dcc9d0be: Pulling fs layer
2023-01-11T21:17:20.8483565Z 90d8f9bbe048: Pulling fs layer
2023-01-11T21:17:20.8484118Z eedfbaa04e4f: Pulling fs layer
2023-01-11T21:17:20.8484620Z ce4a87d45645: Waiting
2023-01-11T21:17:20.8485063Z a08ab4e0594b: Waiting
2023-01-11T21:17:20.8485520Z 6dc2b05bd224: Waiting
2023-01-11T21:17:20.8486000Z 2f2308643d60: Pulling fs layer
2023-01-11T21:17:20.8486479Z 4053f75740ab: Waiting
2023-01-11T21:17:20.8486963Z c1a92fad2c2c: Pulling fs layer
2023-01-11T21:17:20.8487450Z 87d0ffa55850: Waiting
2023-01-11T21:17:20.8487886Z 57e09105cdfd: Waiting
2023-01-11T21:17:20.8488360Z 47037a50f270: Pulling fs layer
2023-01-11T21:17:20.8488826Z bc41eab7f454: Waiting
2023-01-11T21:17:20.8489274Z fa92f16621a4: Waiting
2023-01-11T21:17:20.8489767Z 1a2fd7b216d7: Pulling fs layer
2023-01-11T21:17:20.8490250Z b8f759fd0191: Waiting
2023-01-11T21:17:20.8490607Z 4cd507bccac2: Waiting
2023-01-11T21:17:20.8490998Z 606761d225e5: Waiting
2023-01-11T21:17:20.8491402Z 0c06be5c20e0: Waiting
2023-01-11T21:17:20.8492085Z 765839304d2e: Pulling fs layer
2023-01-11T21:17:20.8492489Z e51794baeb92: Pulling fs layer
2023-01-11T21:17:20.8492752Z f410dcc9d0be: Waiting
2023-01-11T21:17:20.8493016Z ea4bfeaa0fc7: Pulling fs layer
2023-01-11T21:17:20.8493255Z 2f2308643d60: Waiting
2023-01-11T21:17:20.8493506Z d8065d17513d: Pulling fs layer
2023-01-11T21:17:20.8493761Z c1a92fad2c2c: Waiting
2023-01-11T21:17:20.8494001Z 6d83ca3dedf3: Pulling fs layer
2023-01-11T21:17:20.8494270Z 12ddc57b99eb: Pulling fs layer
2023-01-11T21:17:20.8494536Z b590670d273c: Pulling fs layer
2023-01-11T21:17:20.8494787Z 8afbc57dfec9: Pulling fs layer
2023-01-11T21:17:20.8495040Z 1001f0d2f3d0: Waiting
2023-01-11T21:17:20.8495278Z 90d8f9bbe048: Waiting
2023-01-11T21:17:20.8495510Z 1a2fd7b216d7: Waiting
2023-01-11T21:17:20.8495765Z 29a7c0d5fa4c: Pulling fs layer
2023-01-11T21:17:20.8496017Z d8065d17513d: Waiting
2023-01-11T21:17:20.8496246Z 16825bb02017: Pulling fs layer
2023-01-11T21:17:20.8496494Z e1c655e7ec0e: Waiting
2023-01-11T21:17:20.8496752Z bdf297d7f88c: Pulling fs layer
2023-01-11T21:17:20.8496991Z 12ddc57b99eb: Waiting
2023-01-11T21:17:20.8497242Z 885c12efa4ae: Pulling fs layer
2023-01-11T21:17:20.8497495Z 29a7c0d5fa4c: Waiting
2023-01-11T21:17:20.8497719Z bdf297d7f88c: Waiting
2023-01-11T21:17:20.8497969Z 765839304d2e: Waiting
2023-01-11T21:17:20.8498209Z 8afbc57dfec9: Waiting
2023-01-11T21:17:20.8498425Z 47037a50f270: Waiting
2023-01-11T21:17:20.8498658Z f9f75aaba8d7: Waiting
2023-01-11T21:17:20.8498907Z 28c5689cb975: Pulling fs layer
2023-01-11T21:17:20.8499155Z cca768f96df4: Pulling fs layer
2023-01-11T21:17:20.8499426Z 904b81494b5e: Pulling fs layer
2023-01-11T21:17:20.8499690Z 61eecfa8b34e: Pulling fs layer
2023-01-11T21:17:20.8499934Z 885c12efa4ae: Waiting
2023-01-11T21:17:20.8500187Z 95c1ac011645: Pulling fs layer
2023-01-11T21:17:20.8500434Z 16825bb02017: Waiting
2023-01-11T21:17:20.8500702Z 07cee023724c: Pulling fs layer
2023-01-11T21:17:20.8500940Z cca768f96df4: Waiting
2023-01-11T21:17:20.8501176Z 28c5689cb975: Waiting
2023-01-11T21:17:20.8501416Z 61eecfa8b34e: Waiting
2023-01-11T21:17:20.8501647Z 195d560d8cf6: Pulling fs layer
2023-01-11T21:17:20.8501896Z 904b81494b5e: Waiting
2023-01-11T21:17:20.8502140Z a399389c7f8e: Pulling fs layer
2023-01-11T21:17:20.8502388Z 7447f84b33ef: Pulling fs layer
2023-01-11T21:17:20.8502635Z 195d560d8cf6: Waiting
2023-01-11T21:17:20.8502873Z a399389c7f8e: Waiting
2023-01-11T21:17:20.8503105Z 0d8aeb1421f9: Pulling fs layer
2023-01-11T21:17:20.8503367Z 02048a597c22: Pulling fs layer
2023-01-11T21:17:20.8503628Z 25d615d8a5e2: Pulling fs layer
2023-01-11T21:17:20.8503864Z e51794baeb92: Waiting
2023-01-11T21:17:20.8504112Z 09d400b86049: Pulling fs layer
2023-01-11T21:17:20.8504361Z 0d8aeb1421f9: Waiting
2023-01-11T21:17:20.8504661Z 7447f84b33ef: Waiting
2023-01-11T21:17:20.8504892Z 02048a597c22: Waiting
2023-01-11T21:17:20.8505117Z 09d400b86049: Waiting
2023-01-11T21:17:20.9867269Z 4542784317be: Download complete
2023-01-11T21:17:21.1383126Z 4053f75740ab: Verifying Checksum
2023-01-11T21:17:21.1383471Z 4053f75740ab: Download complete
2023-01-11T21:17:21.1713287Z fb668870d8a7: Verifying Checksum
2023-01-11T21:17:21.1713574Z fb668870d8a7: Download complete
2023-01-11T21:17:21.2271272Z 57e09105cdfd: Verifying Checksum
2023-01-11T21:17:21.2271617Z 57e09105cdfd: Download complete
2023-01-11T21:17:21.3265788Z 69473a703fb4: Verifying Checksum
2023-01-11T21:17:21.3266156Z 69473a703fb4: Download complete
2023-01-11T21:17:21.3749395Z e0bec5df5af5: Verifying Checksum
2023-01-11T21:17:21.3749709Z e0bec5df5af5: Download complete
2023-01-11T21:17:21.4150352Z a08ab4e0594b: Download complete
2023-01-11T21:17:21.4671771Z 4cd507bccac2: Verifying Checksum
2023-01-11T21:17:21.4672062Z 4cd507bccac2: Download complete
2023-01-11T21:17:21.5469030Z 6dc2b05bd224: Verifying Checksum
2023-01-11T21:17:21.5469379Z 6dc2b05bd224: Download complete
2023-01-11T21:17:21.6452882Z ce4a87d45645: Verifying Checksum
2023-01-11T21:17:21.6453481Z ce4a87d45645: Download complete
2023-01-11T21:17:21.9356471Z fb668870d8a7: Pull complete
2023-01-11T21:17:22.2119139Z 4542784317be: Pull complete
2023-01-11T21:17:23.0909828Z e0bec5df5af5: Pull complete
2023-01-11T21:17:23.1959957Z 4053f75740ab: Pull complete
2023-01-11T21:17:23.3018808Z 57e09105cdfd: Pull complete
2023-01-11T21:17:23.6993372Z 41860ea59b6c: Verifying Checksum
2023-01-11T21:17:23.6993703Z 41860ea59b6c: Download complete
2023-01-11T21:17:23.7846110Z 87d0ffa55850: Verifying Checksum
2023-01-11T21:17:23.7846454Z 87d0ffa55850: Download complete
2023-01-11T21:17:23.8617417Z f9f75aaba8d7: Verifying Checksum
2023-01-11T21:17:23.8617752Z f9f75aaba8d7: Download complete
2023-01-11T21:17:23.9808983Z 0c06be5c20e0: Verifying Checksum
2023-01-11T21:17:23.9809330Z 0c06be5c20e0: Download complete
2023-01-11T21:17:24.7164060Z d23c0a07b67c: Verifying Checksum
2023-01-11T21:17:24.7164426Z d23c0a07b67c: Download complete
2023-01-11T21:17:24.7920807Z 1001f0d2f3d0: Verifying Checksum
2023-01-11T21:17:24.7921079Z 1001f0d2f3d0: Download complete
2023-01-11T21:17:24.8819147Z e1c655e7ec0e: Verifying Checksum
2023-01-11T21:17:24.8819486Z e1c655e7ec0e: Download complete
2023-01-11T21:17:32.1222234Z 606761d225e5: Download complete
2023-01-11T21:17:32.2022279Z bc41eab7f454: Download complete
2023-01-11T21:17:32.2856733Z b8f759fd0191: Download complete
2023-01-11T21:17:32.3680022Z f410dcc9d0be: Download complete
2023-01-11T21:17:32.4754917Z 90d8f9bbe048: Verifying Checksum
2023-01-11T21:17:32.4755249Z 90d8f9bbe048: Download complete
2023-01-11T21:17:32.5466847Z eedfbaa04e4f: Verifying Checksum
2023-01-11T21:17:32.5467324Z eedfbaa04e4f: Download complete
2023-01-11T21:17:32.6080998Z 2f2308643d60: Verifying Checksum
2023-01-11T21:17:32.6081319Z 2f2308643d60: Download complete
2023-01-11T21:17:33.5450976Z c1a92fad2c2c: Verifying Checksum
2023-01-11T21:17:33.5451352Z c1a92fad2c2c: Download complete
2023-01-11T21:17:33.6268292Z 47037a50f270: Verifying Checksum
2023-01-11T21:17:33.6268669Z 47037a50f270: Download complete
2023-01-11T21:17:33.7284621Z 1a2fd7b216d7: Verifying Checksum
2023-01-11T21:17:33.7285010Z 1a2fd7b216d7: Download complete
2023-01-11T21:17:33.8275306Z 765839304d2e: Verifying Checksum
2023-01-11T21:17:33.8275810Z 765839304d2e: Download complete
2023-01-11T21:17:33.9073258Z e51794baeb92: Download complete
2023-01-11T21:17:33.9749719Z ea4bfeaa0fc7: Download complete
2023-01-11T21:17:35.5864601Z fa92f16621a4: Verifying Checksum
2023-01-11T21:17:35.5865224Z fa92f16621a4: Download complete
2023-01-11T21:17:35.6718473Z 6d83ca3dedf3: Verifying Checksum
2023-01-11T21:17:35.6718836Z 6d83ca3dedf3: Download complete
2023-01-11T21:17:35.7570847Z 12ddc57b99eb: Verifying Checksum
2023-01-11T21:17:35.7571299Z 12ddc57b99eb: Download complete
2023-01-11T21:17:35.9763210Z d8065d17513d: Verifying Checksum
2023-01-11T21:17:35.9763811Z d8065d17513d: Download complete
2023-01-11T21:17:36.0721329Z 8afbc57dfec9: Verifying Checksum
2023-01-11T21:17:36.0721880Z 8afbc57dfec9: Download complete
2023-01-11T21:17:36.1597039Z 29a7c0d5fa4c: Verifying Checksum
2023-01-11T21:17:36.1597377Z 29a7c0d5fa4c: Download complete
2023-01-11T21:17:36.2160699Z b590670d273c: Verifying Checksum
2023-01-11T21:17:36.2161009Z b590670d273c: Download complete
2023-01-11T21:17:36.3015035Z bdf297d7f88c: Verifying Checksum
2023-01-11T21:17:36.3015387Z bdf297d7f88c: Download complete
2023-01-11T21:17:36.4108982Z 16825bb02017: Verifying Checksum
2023-01-11T21:17:36.4109312Z 16825bb02017: Download complete
2023-01-11T21:17:36.5077681Z 28c5689cb975: Download complete
2023-01-11T21:17:36.5870146Z cca768f96df4: Verifying Checksum
2023-01-11T21:17:36.5870534Z cca768f96df4: Download complete
2023-01-11T21:17:36.8461142Z 885c12efa4ae: Verifying Checksum
2023-01-11T21:17:36.8461511Z 885c12efa4ae: Download complete
2023-01-11T21:17:36.9171273Z 61eecfa8b34e: Verifying Checksum
2023-01-11T21:17:36.9171668Z 61eecfa8b34e: Download complete
2023-01-11T21:17:37.0142358Z 95c1ac011645: Verifying Checksum
2023-01-11T21:17:37.0142711Z 95c1ac011645: Download complete
2023-01-11T21:17:37.0864456Z 07cee023724c: Download complete
2023-01-11T21:17:37.1682502Z 195d560d8cf6: Verifying Checksum
2023-01-11T21:17:37.1683047Z 195d560d8cf6: Download complete
2023-01-11T21:17:37.4169159Z a399389c7f8e: Verifying Checksum
2023-01-11T21:17:37.4169805Z a399389c7f8e: Download complete
2023-01-11T21:17:37.5117936Z 7447f84b33ef: Verifying Checksum
2023-01-11T21:17:37.5118434Z 7447f84b33ef: Download complete
2023-01-11T21:17:38.1103872Z 0d8aeb1421f9: Verifying Checksum
2023-01-11T21:17:38.1104251Z 0d8aeb1421f9: Download complete
2023-01-11T21:17:38.2217111Z 02048a597c22: Verifying Checksum
2023-01-11T21:17:38.2217738Z 02048a597c22: Download complete
2023-01-11T21:17:39.3974794Z 904b81494b5e: Verifying Checksum
2023-01-11T21:17:39.3975410Z 904b81494b5e: Download complete
2023-01-11T21:17:39.4859581Z 09d400b86049: Verifying Checksum
2023-01-11T21:17:39.4859903Z 09d400b86049: Download complete
2023-01-11T21:17:45.6026625Z 606761d225e5: Pull complete
2023-01-11T21:17:46.1873139Z 69473a703fb4: Pull complete
2023-01-11T21:17:46.3246251Z a08ab4e0594b: Pull complete
2023-01-11T21:17:46.4807891Z 4cd507bccac2: Pull complete
2023-01-11T21:18:08.2963235Z fa92f16621a4: Pull complete
2023-01-11T21:18:10.8329456Z 6dc2b05bd224: Pull complete
2023-01-11T21:18:12.7050964Z ce4a87d45645: Pull complete
2023-01-11T21:18:20.9204193Z 41860ea59b6c: Pull complete
2023-01-11T21:18:22.8300727Z 87d0ffa55850: Pull complete
2023-01-11T21:18:24.7747834Z f9f75aaba8d7: Pull complete
2023-01-11T21:18:25.2083167Z a11b4b5fd784: Download complete
2023-01-11T21:18:26.9450040Z 0c06be5c20e0: Pull complete
2023-01-11T21:18:31.0042538Z d23c0a07b67c: Pull complete
2023-01-11T21:18:33.4291694Z 1001f0d2f3d0: Pull complete
2023-01-11T21:18:35.3397212Z e1c655e7ec0e: Pull complete
2023-01-11T21:18:56.5595127Z 25d615d8a5e2: Verifying Checksum
2023-01-11T21:18:56.5597659Z 25d615d8a5e2: Download complete
2023-01-11T21:19:08.6554867Z a11b4b5fd784: Pull complete
2023-01-11T21:19:10.9339915Z bc41eab7f454: Pull complete
2023-01-11T21:19:12.7740016Z b8f759fd0191: Pull complete
2023-01-11T21:19:14.7769799Z f410dcc9d0be: Pull complete
2023-01-11T21:19:16.6703432Z 90d8f9bbe048: Pull complete
2023-01-11T21:19:18.7788849Z eedfbaa04e4f: Pull complete
2023-01-11T21:19:20.9183904Z 2f2308643d60: Pull complete
2023-01-11T21:19:25.1069123Z c1a92fad2c2c: Pull complete
2023-01-11T21:19:27.1372431Z 47037a50f270: Pull complete
2023-01-11T21:19:29.8412268Z 1a2fd7b216d7: Pull complete
2023-01-11T21:19:32.4340541Z 765839304d2e: Pull complete
2023-01-11T21:19:36.3457197Z e51794baeb92: Pull complete
2023-01-11T21:19:38.6793798Z ea4bfeaa0fc7: Pull complete
2023-01-11T21:19:47.1358009Z d8065d17513d: Pull complete
2023-01-11T21:19:49.3270406Z 6d83ca3dedf3: Pull complete
2023-01-11T21:19:51.1570053Z 12ddc57b99eb: Pull complete
2023-01-11T21:19:52.0738148Z b590670d273c: Pull complete
2023-01-11T21:19:52.1692772Z 8afbc57dfec9: Pull complete
2023-01-11T21:19:52.2653691Z 29a7c0d5fa4c: Pull complete
2023-01-11T21:19:52.6832714Z 16825bb02017: Pull complete
2023-01-11T21:19:52.7854740Z bdf297d7f88c: Pull complete
2023-01-11T21:19:54.1987623Z 885c12efa4ae: Pull complete
2023-01-11T21:19:54.3058132Z 28c5689cb975: Pull complete
2023-01-11T21:19:54.4023682Z cca768f96df4: Pull complete
2023-01-11T21:19:59.6066428Z 904b81494b5e: Pull complete
2023-01-11T21:19:59.7292401Z 61eecfa8b34e: Pull complete
2023-01-11T21:19:59.8393355Z 95c1ac011645: Pull complete
2023-01-11T21:20:00.3453883Z 07cee023724c: Pull complete
2023-01-11T21:20:00.4611552Z 195d560d8cf6: Pull complete
2023-01-11T21:20:01.2288549Z a399389c7f8e: Pull complete
2023-01-11T21:20:01.3368091Z 7447f84b33ef: Pull complete
2023-01-11T21:20:03.2498508Z 0d8aeb1421f9: Pull complete
2023-01-11T21:20:03.3483810Z 02048a597c22: Pull complete
2023-01-11T21:20:33.9965681Z 25d615d8a5e2: Pull complete
2023-01-11T21:20:36.0179035Z 09d400b86049: Pull complete
2023-01-11T21:20:37.3321018Z Digest: sha256:0da23f4faf0ce20770149c4a783e08eaa91c07112511dc5511c77937c66edb24
2023-01-11T21:20:37.8331401Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T21:20:38.1144893Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T21:20:38.1247361Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main
2023-01-11T21:20:38.1247691Z with:
2023-01-11T21:20:38.1247928Z   driver-version: 515.76
2023-01-11T21:20:38.1248166Z env:
2023-01-11T21:20:38.1248382Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:20:38.1248640Z ##[endgroup]
2023-01-11T21:20:38.2721018Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2023-01-11T21:20:38.2721315Z with:
2023-01-11T21:20:38.2721545Z   timeout_minutes: 10
2023-01-11T21:20:38.2721787Z   max_attempts: 3
2023-01-11T21:20:38.2728659Z   command: # Is it disgusting to have a full shell script here in this github action? Sure
# But is it the best way to make it so that this action relies on nothing else? Absolutely
set -eou pipefail

DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID)
DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run"
YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo"

install_nvidia_docker2_amzn2() {
    (
        set -x
        # Needed for yum-config-manager
        sudo yum install -y yum-utils
        sudo yum-config-manager --add-repo "${YUM_REPO_URL}"
        sudo yum install -y nvidia-docker2
        sudo systemctl restart docker
    )
}

install_nvidia_docker2_ubuntu20() {
    (
        set -x
        sudo apt-get install -y nvidia-docker2
        sudo systemctl restart docker
    )
}

pre_install_nvidia_driver_amzn2() {
    (
        # Purge any nvidia driver installed from RHEL repo
        sudo yum remove -y nvidia-driver-latest-dkms
    )
}

install_nvidia_driver_common() {
    (
        # Try to gather more information about the runner and its existing NVIDIA driver if any
        echo "Before installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        HAS_NVIDIA_DRIVER=0
        # Check if NVIDIA driver has already been installed
        if [ -x "$(command -v nvidia-smi)" ]; then
            set +e
            # The driver exists, check its version next. Also check only the first GPU if there are more than one of them
            # so that the same driver version is not print over multiple lines
            INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
            NVIDIA_SMI_STATUS=$?

            if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing"
            elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing"
            else
                HAS_NVIDIA_DRIVER=1
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation"
            fi
            set -e
        fi

        if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then
            # CAUTION: this may need to be updated in future
            if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then
                  sudo yum groupinstall -y "Development Tools"
                  # ensure our kernel install is the same as our underlying kernel,
                  # groupinstall "Development Tools" has a habit of mismatching kernel headers
                  sudo yum install -y "kernel-devel-uname-r == $(uname -r)"
                  sudo modprobe backlight
            fi
            sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN"

            set +e
            sudo /bin/bash /tmp/nvidia_driver -s --no-drm
            NVIDIA_INSTALLATION_STATUS=$?

            RESET_GPU=0
            if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then
                sudo cat /var/log/nvidia-installer.log
                # Fail to install NVIDIA driver, try to reset the GPU
                RESET_GPU=1
            elif [ -x "$(command -v nvidia-smi)" ]; then
                # Check again if nvidia-smi works even if the driver installation completes successfully
                INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
                NVIDIA_SMI_STATUS=$?

                if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                    RESET_GPU=1
                fi
            fi

            if [ "$RESET_GPU" -eq 1 ]; then
                NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1)
                # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this
                # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388
                for PCI_ID in $NVIDIA_DEVICES; do
                    DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable)

                    echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)"
                    # This requires sudo permission of course
                    echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset
                    sleep 1
                done
            fi

            sudo rm -fv /tmp/nvidia_driver
            set -e
        fi
    )
}

post_install_nvidia_driver_common() {
    (
        sudo modprobe nvidia || true
        echo "After installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        (
            set +e
            nvidia-smi
            NVIDIA_SMI_STATUS=$?

            # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285
            if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then
                echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}"
            else
                echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}"
                exit ${NVIDIA_SMI_STATUS}
            fi
            set -e
        )
    )
}

install_nvidia_driver_amzn2() {
    (
        set -x
        pre_install_nvidia_driver_amzn2
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

install_nvidia_driver_ubuntu20() {
    (
        set -x
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

echo "== Installing nvidia driver ${DRIVER_FN} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_driver_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_driver_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac

# Install container toolkit based on distribution
echo "== Installing nvidia container toolkit for ${DISTRIBUTION} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_docker2_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_docker2_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac
echo "GPU_FLAG=--gpus all" >> "${GITHUB_ENV}"

2023-01-11T21:20:38.2735734Z   retry_wait_seconds: 10
2023-01-11T21:20:38.2736009Z   polling_interval_seconds: 1
2023-01-11T21:20:38.2736274Z   warning_on_retry: true
2023-01-11T21:20:38.2736511Z   continue_on_error: false
2023-01-11T21:20:38.2736747Z env:
2023-01-11T21:20:38.2736977Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:20:38.2737222Z   DRIVER_VERSION: 515.76
2023-01-11T21:20:38.2737469Z ##[endgroup]
2023-01-11T21:20:38.3416438Z == Installing nvidia driver NVIDIA-Linux-x86_64-515.76.run ==
2023-01-11T21:20:38.3418490Z + pre_install_nvidia_driver_amzn2
2023-01-11T21:20:38.3422138Z + sudo yum remove -y nvidia-driver-latest-dkms
2023-01-11T21:20:38.9149749Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2023-01-11T21:20:38.9730447Z No Match for argument: nvidia-driver-latest-dkms
2023-01-11T21:20:39.0108301Z No Packages marked for removal
2023-01-11T21:20:39.0269326Z + install_nvidia_driver_common
2023-01-11T21:20:39.0273632Z + echo 'Before installing NVIDIA driver'
2023-01-11T21:20:39.0275222Z + lspci
2023-01-11T21:20:39.0277140Z Before installing NVIDIA driver
2023-01-11T21:20:39.0465626Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
2023-01-11T21:20:39.0466072Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2023-01-11T21:20:39.0466475Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
2023-01-11T21:20:39.0466852Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
2023-01-11T21:20:39.0469564Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446
2023-01-11T21:20:39.0469991Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2023-01-11T21:20:39.0470433Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2023-01-11T21:20:39.0470856Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2023-01-11T21:20:39.0471260Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)
2023-01-11T21:20:39.0471575Z + lsmod
2023-01-11T21:20:39.0488541Z Module                  Size  Used by
2023-01-11T21:20:39.0488839Z nvidia_modeset       1142784  0
2023-01-11T21:20:39.0489111Z nvidia_uvm           1269760  0
2023-01-11T21:20:39.0489370Z veth                   16384  0
2023-01-11T21:20:39.0489659Z nvidia              40808448  15 nvidia_uvm,nvidia_modeset
2023-01-11T21:20:39.0489928Z drm                   425984  1 nvidia
2023-01-11T21:20:39.0490197Z i2c_core               77824  2 nvidia,drm
2023-01-11T21:20:39.0490483Z backlight              16384  1 nvidia_modeset
2023-01-11T21:20:39.0490739Z xt_conntrack           16384  1
2023-01-11T21:20:39.0491005Z ipt_MASQUERADE         16384  1
2023-01-11T21:20:39.0491305Z nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
2023-01-11T21:20:39.0491585Z nf_conntrack_netlink    49152  0
2023-01-11T21:20:39.0491880Z nfnetlink              16384  2 nf_conntrack_netlink
2023-01-11T21:20:39.0492169Z xfrm_user              45056  1
2023-01-11T21:20:39.0492439Z xfrm_algo              16384  1 xfrm_user
2023-01-11T21:20:39.0492687Z xt_addrtype            16384  2
2023-01-11T21:20:39.0492951Z iptable_filter         16384  1
2023-01-11T21:20:39.0493213Z iptable_nat            16384  1
2023-01-11T21:20:39.0493455Z nf_conntrack_ipv4      16384  3
2023-01-11T21:20:39.0493744Z nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
2023-01-11T21:20:39.0494049Z nf_nat_ipv4            16384  1 iptable_nat
2023-01-11T21:20:39.0494350Z nf_nat                 36864  2 nf_nat_masquerade_ipv4,nf_nat_ipv4
2023-01-11T21:20:39.0494830Z nf_conntrack          155648  7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink
2023-01-11T21:20:39.0495449Z br_netfilter           24576  0
2023-01-11T21:20:39.0495726Z bridge                172032  1 br_netfilter
2023-01-11T21:20:39.0495978Z stp                    16384  1 bridge
2023-01-11T21:20:39.0496243Z llc                    16384  2 bridge,stp
2023-01-11T21:20:39.0496504Z overlay                86016  0
2023-01-11T21:20:39.0496753Z sunrpc                393216  1
2023-01-11T21:20:39.0497010Z dm_mirror              28672  0
2023-01-11T21:20:39.0497278Z dm_region_hash         20480  1 dm_mirror
2023-01-11T21:20:39.0497563Z dm_log                 20480  2 dm_region_hash,dm_mirror
2023-01-11T21:20:39.0497868Z dm_mod                143360  2 dm_log,dm_mirror
2023-01-11T21:20:39.0498136Z dax                    69632  1 dm_mod
2023-01-11T21:20:39.0498372Z sb_edac                24576  0
2023-01-11T21:20:39.0498625Z crc32_pclmul           16384  0
2023-01-11T21:20:39.0498889Z ghash_clmulni_intel    16384  0
2023-01-11T21:20:39.0499130Z pcbc                   16384  0
2023-01-11T21:20:39.0499383Z ata_piix               36864  0
2023-01-11T21:20:39.0499635Z aesni_intel           188416  0
2023-01-11T21:20:39.0499885Z mousedev               24576  0
2023-01-11T21:20:39.0500125Z libata                266240  1 ata_piix
2023-01-11T21:20:39.0500405Z aes_x86_64             20480  1 aesni_intel
2023-01-11T21:20:39.0500684Z crypto_simd            16384  1 aesni_intel
2023-01-11T21:20:39.0501067Z pcc_cpufreq            16384  0
2023-01-11T21:20:39.0501363Z glue_helper            16384  1 aesni_intel
2023-01-11T21:20:39.0501690Z cryptd                 28672  3 crypto_simd,ghash_clmulni_intel,aesni_intel
2023-01-11T21:20:39.0502009Z scsi_mod              245760  1 libata
2023-01-11T21:20:39.0502251Z psmouse                32768  0
2023-01-11T21:20:39.0502501Z evdev                  20480  3
2023-01-11T21:20:39.0502744Z button                 16384  0
2023-01-11T21:20:39.0502970Z ena                   114688  0
2023-01-11T21:20:39.0503213Z xen_blkfront           49152  2
2023-01-11T21:20:39.0503473Z crc32c_intel           24576  0
2023-01-11T21:20:39.0503706Z autofs4                49152  2
2023-01-11T21:20:39.0503946Z + modinfo nvidia
2023-01-11T21:20:39.0504439Z filename:       /lib/modules/4.14.252-195.483.amzn2.x86_64/kernel/drivers/video/nvidia.ko
2023-01-11T21:20:39.0504765Z firmware:       nvidia/515.76/gsp.bin
2023-01-11T21:20:39.0505095Z alias:          char-major-195-*
2023-01-11T21:20:39.0505362Z version:        515.76
2023-01-11T21:20:39.0505598Z supported:      external
2023-01-11T21:20:39.0505850Z license:        NVIDIA
2023-01-11T21:20:39.0506118Z srcversion:     51FD9DD90150B35351AFFBB
2023-01-11T21:20:39.0506410Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2023-01-11T21:20:39.0506718Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2023-01-11T21:20:39.0507021Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2023-01-11T21:20:39.0507333Z depends:        i2c-core,drm
2023-01-11T21:20:39.0507593Z retpoline:      Y
2023-01-11T21:20:39.0507832Z name:           nvidia
2023-01-11T21:20:39.0508212Z vermagic:       4.14.252-195.483.amzn2.x86_64 SMP mod_unload modversions 
2023-01-11T21:20:39.0508585Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2023-01-11T21:20:39.0508972Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2023-01-11T21:20:39.0509333Z parm:           NVreg_ResmanDebugLevel:int
2023-01-11T21:20:39.0509611Z parm:           NVreg_RmLogonRC:int
2023-01-11T21:20:39.0509914Z parm:           NVreg_ModifyDeviceFiles:int
2023-01-11T21:20:39.0510215Z parm:           NVreg_DeviceFileUID:int
2023-01-11T21:20:39.0510487Z parm:           NVreg_DeviceFileGID:int
2023-01-11T21:20:39.0510786Z parm:           NVreg_DeviceFileMode:int
2023-01-11T21:20:39.0511141Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2023-01-11T21:20:39.0511493Z parm:           NVreg_UsePageAttributeTable:int
2023-01-11T21:20:39.0511808Z parm:           NVreg_EnablePCIeGen3:int
2023-01-11T21:20:39.0512822Z parm:           NVreg_EnableMSI:int
2023-01-11T21:20:39.0513216Z parm:           NVreg_TCEBypassMode:int
2023-01-11T21:20:39.0513511Z parm:           NVreg_EnableStreamMemOPs:int
2023-01-11T21:20:39.0513864Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2023-01-11T21:20:39.0514248Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2023-01-11T21:20:39.0514603Z parm:           NVreg_EnableS0ixPowerManagement:int
2023-01-11T21:20:39.0515008Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2023-01-11T21:20:39.0515400Z parm:           NVreg_DynamicPowerManagement:int
2023-01-11T21:20:39.0515790Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2023-01-11T21:20:39.0516182Z parm:           NVreg_EnableGpuFirmware:int
2023-01-11T21:20:39.0516506Z parm:           NVreg_EnableGpuFirmwareLogs:int
2023-01-11T21:20:39.0516858Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2023-01-11T21:20:39.0517202Z parm:           NVreg_EnableUserNUMAManagement:int
2023-01-11T21:20:39.0517525Z parm:           NVreg_MemoryPoolSize:int
2023-01-11T21:20:39.0517844Z parm:           NVreg_KMallocHeapMaxSize:int
2023-01-11T21:20:39.0518146Z parm:           NVreg_VMallocHeapMaxSize:int
2023-01-11T21:20:39.0518458Z parm:           NVreg_IgnoreMMIOCheck:int
2023-01-11T21:20:39.0518752Z parm:           NVreg_NvLinkDisable:int
2023-01-11T21:20:39.0519077Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2023-01-11T21:20:39.0519485Z parm:           NVreg_RegisterPCIDriver:int
2023-01-11T21:20:39.0519819Z parm:           NVreg_EnableDbgBreakpoint:int
2023-01-11T21:20:39.0520117Z parm:           NVreg_RegistryDwords:charp
2023-01-11T21:20:39.0520447Z parm:           NVreg_RegistryDwordsPerDevice:charp
2023-01-11T21:20:39.0520754Z parm:           NVreg_RmMsg:charp
2023-01-11T21:20:39.0521020Z parm:           NVreg_GpuBlacklist:charp
2023-01-11T21:20:39.0521334Z parm:           NVreg_TemporaryFilePath:charp
2023-01-11T21:20:39.0521645Z parm:           NVreg_ExcludedGpus:charp
2023-01-11T21:20:39.0521948Z parm:           NVreg_DmaRemapPeerMmio:int
2023-01-11T21:20:39.0522240Z parm:           rm_firmware_active:charp
2023-01-11T21:20:39.0522513Z + HAS_NVIDIA_DRIVER=0
2023-01-11T21:20:39.0522832Z ++ command -v nvidia-smi
2023-01-11T21:20:39.0523131Z + '[' -x /usr/bin/nvidia-smi ']'
2023-01-11T21:20:39.0523375Z + set +e
2023-01-11T21:20:39.0523764Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0
2023-01-11T21:20:39.4710851Z + INSTALLED_DRIVER_VERSION=515.76
2023-01-11T21:20:39.4711460Z + NVIDIA_SMI_STATUS=0
2023-01-11T21:20:39.4712762Z + '[' 0 -ne 0 ']'
2023-01-11T21:20:39.4713174Z + '[' 515.76 '!=' 515.76 ']'
2023-01-11T21:20:39.4713422Z + HAS_NVIDIA_DRIVER=1
2023-01-11T21:20:39.4713887Z + echo 'NVIDIA driver (515.76) has already been installed. Skipping NVIDIA driver installation'
2023-01-11T21:20:39.4714228Z + set -e
2023-01-11T21:20:39.4714494Z + '[' 1 -eq 0 ']'
2023-01-11T21:20:39.4714835Z NVIDIA driver (515.76) has already been installed. Skipping NVIDIA driver installation
2023-01-11T21:20:39.4715195Z + post_install_nvidia_driver_common
2023-01-11T21:20:39.4717449Z + sudo modprobe nvidia
2023-01-11T21:20:39.4994832Z + echo 'After installing NVIDIA driver'
2023-01-11T21:20:39.4995314Z + lspci
2023-01-11T21:20:39.4996040Z After installing NVIDIA driver
2023-01-11T21:20:39.5213425Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
2023-01-11T21:20:39.5214265Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2023-01-11T21:20:39.5215140Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
2023-01-11T21:20:39.5215559Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
2023-01-11T21:20:39.5215924Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446
2023-01-11T21:20:39.5216307Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2023-01-11T21:20:39.5216709Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2023-01-11T21:20:39.5217131Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2023-01-11T21:20:39.5217733Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)
2023-01-11T21:20:39.5218054Z + lsmod
2023-01-11T21:20:39.5236237Z Module                  Size  Used by
2023-01-11T21:20:39.5236708Z nvidia_modeset       1142784  0
2023-01-11T21:20:39.5237137Z nvidia_uvm           1269760  0
2023-01-11T21:20:39.5237456Z veth                   16384  0
2023-01-11T21:20:39.5237997Z nvidia              40808448  27 nvidia_uvm,nvidia_modeset
2023-01-11T21:20:39.5238593Z drm                   425984  1 nvidia
2023-01-11T21:20:39.5239104Z i2c_core               77824  2 nvidia,drm
2023-01-11T21:20:39.5239724Z backlight              16384  1 nvidia_modeset
2023-01-11T21:20:39.5240323Z xt_conntrack           16384  1
2023-01-11T21:20:39.5240711Z ipt_MASQUERADE         16384  1
2023-01-11T21:20:39.5240993Z nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
2023-01-11T21:20:39.5241319Z nf_conntrack_netlink    49152  0
2023-01-11T21:20:39.5241624Z nfnetlink              16384  2 nf_conntrack_netlink
2023-01-11T21:20:39.5241885Z xfrm_user              45056  1
2023-01-11T21:20:39.5242149Z xfrm_algo              16384  1 xfrm_user
2023-01-11T21:20:39.5242415Z xt_addrtype            16384  2
2023-01-11T21:20:39.5242659Z iptable_filter         16384  1
2023-01-11T21:20:39.5242918Z iptable_nat            16384  1
2023-01-11T21:20:39.5243178Z nf_conntrack_ipv4      16384  3
2023-01-11T21:20:39.5243605Z nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
2023-01-11T21:20:39.5243907Z nf_nat_ipv4            16384  1 iptable_nat
2023-01-11T21:20:39.5244223Z nf_nat                 36864  2 nf_nat_masquerade_ipv4,nf_nat_ipv4
2023-01-11T21:20:39.5244687Z nf_conntrack          155648  7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink
2023-01-11T21:20:39.5245072Z br_netfilter           24576  0
2023-01-11T21:20:39.5245344Z bridge                172032  1 br_netfilter
2023-01-11T21:20:39.5245613Z stp                    16384  1 bridge
2023-01-11T21:20:39.5245863Z llc                    16384  2 bridge,stp
2023-01-11T21:20:39.5246120Z overlay                86016  0
2023-01-11T21:20:39.5246367Z sunrpc                393216  1
2023-01-11T21:20:39.5246600Z dm_mirror              28672  0
2023-01-11T21:20:39.5246869Z dm_region_hash         20480  1 dm_mirror
2023-01-11T21:20:39.5247175Z dm_log                 20480  2 dm_region_hash,dm_mirror
2023-01-11T21:20:39.5247477Z dm_mod                143360  2 dm_log,dm_mirror
2023-01-11T21:20:39.5247729Z dax                    69632  1 dm_mod
2023-01-11T21:20:39.5248050Z sb_edac                24576  0
2023-01-11T21:20:39.5248540Z crc32_pclmul           16384  0
2023-01-11T21:20:39.5249007Z ghash_clmulni_intel    16384  0
2023-01-11T21:20:39.5249288Z pcbc                   16384  0
2023-01-11T21:20:39.5249743Z ata_piix               36864  0
2023-01-11T21:20:39.5250049Z aesni_intel           188416  0
2023-01-11T21:20:39.5250584Z mousedev               24576  0
2023-01-11T21:20:39.5251155Z libata                266240  1 ata_piix
2023-01-11T21:20:39.5251498Z aes_x86_64             20480  1 aesni_intel
2023-01-11T21:20:39.5251778Z crypto_simd            16384  1 aesni_intel
2023-01-11T21:20:39.5252180Z pcc_cpufreq            16384  0
2023-01-11T21:20:39.5252722Z glue_helper            16384  1 aesni_intel
2023-01-11T21:20:39.5253463Z cryptd                 28672  3 crypto_simd,ghash_clmulni_intel,aesni_intel
2023-01-11T21:20:39.5254176Z scsi_mod              245760  1 libata
2023-01-11T21:20:39.5254650Z psmouse                32768  0
2023-01-11T21:20:39.5254909Z evdev                  20480  3
2023-01-11T21:20:39.5255153Z button                 16384  0
2023-01-11T21:20:39.5255393Z ena                   114688  0
2023-01-11T21:20:39.5255621Z xen_blkfront           49152  2
2023-01-11T21:20:39.5255993Z crc32c_intel           24576  0
2023-01-11T21:20:39.5256340Z autofs4                49152  2
2023-01-11T21:20:39.5256607Z + modinfo nvidia
2023-01-11T21:20:39.5257396Z filename:       /lib/modules/4.14.252-195.483.amzn2.x86_64/kernel/drivers/video/nvidia.ko
2023-01-11T21:20:39.5257769Z firmware:       nvidia/515.76/gsp.bin
2023-01-11T21:20:39.5258205Z alias:          char-major-195-*
2023-01-11T21:20:39.5258518Z version:        515.76
2023-01-11T21:20:39.5258777Z supported:      external
2023-01-11T21:20:39.5259119Z license:        NVIDIA
2023-01-11T21:20:39.5259450Z srcversion:     51FD9DD90150B35351AFFBB
2023-01-11T21:20:39.5259772Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2023-01-11T21:20:39.5260131Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2023-01-11T21:20:39.5260484Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2023-01-11T21:20:39.5260909Z depends:        i2c-core,drm
2023-01-11T21:20:39.5261176Z retpoline:      Y
2023-01-11T21:20:39.5261481Z name:           nvidia
2023-01-11T21:20:39.5261927Z vermagic:       4.14.252-195.483.amzn2.x86_64 SMP mod_unload modversions 
2023-01-11T21:20:39.5262313Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2023-01-11T21:20:39.5262811Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2023-01-11T21:20:39.5263224Z parm:           NVreg_ResmanDebugLevel:int
2023-01-11T21:20:39.5263527Z parm:           NVreg_RmLogonRC:int
2023-01-11T21:20:39.5263881Z parm:           NVreg_ModifyDeviceFiles:int
2023-01-11T21:20:39.5264277Z parm:           NVreg_DeviceFileUID:int
2023-01-11T21:20:39.5264639Z parm:           NVreg_DeviceFileGID:int
2023-01-11T21:20:39.5265011Z parm:           NVreg_DeviceFileMode:int
2023-01-11T21:20:39.5265419Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2023-01-11T21:20:39.5265856Z parm:           NVreg_UsePageAttributeTable:int
2023-01-11T21:20:39.5266225Z parm:           NVreg_EnablePCIeGen3:int
2023-01-11T21:20:39.5266567Z parm:           NVreg_EnableMSI:int
2023-01-11T21:20:39.5266900Z parm:           NVreg_TCEBypassMode:int
2023-01-11T21:20:39.5267233Z parm:           NVreg_EnableStreamMemOPs:int
2023-01-11T21:20:39.5267639Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2023-01-11T21:20:39.5268135Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2023-01-11T21:20:39.5268525Z parm:           NVreg_EnableS0ixPowerManagement:int
2023-01-11T21:20:39.5269004Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2023-01-11T21:20:39.5269469Z parm:           NVreg_DynamicPowerManagement:int
2023-01-11T21:20:39.5269902Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2023-01-11T21:20:39.5270402Z parm:           NVreg_EnableGpuFirmware:int
2023-01-11T21:20:39.5270803Z parm:           NVreg_EnableGpuFirmwareLogs:int
2023-01-11T21:20:39.5271180Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2023-01-11T21:20:39.5271624Z parm:           NVreg_EnableUserNUMAManagement:int
2023-01-11T21:20:39.5272574Z parm:           NVreg_MemoryPoolSize:int
2023-01-11T21:20:39.5273002Z parm:           NVreg_KMallocHeapMaxSize:int
2023-01-11T21:20:39.5297268Z parm:           NVreg_VMallocHeapMaxSize:int
2023-01-11T21:20:39.5297626Z parm:           NVreg_IgnoreMMIOCheck:int
2023-01-11T21:20:39.5297936Z parm:           NVreg_NvLinkDisable:int
2023-01-11T21:20:39.5298268Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2023-01-11T21:20:39.5298629Z parm:           NVreg_RegisterPCIDriver:int
2023-01-11T21:20:39.5298959Z parm:           NVreg_EnableDbgBreakpoint:int
2023-01-11T21:20:39.5299268Z parm:           NVreg_RegistryDwords:charp
2023-01-11T21:20:39.5299614Z parm:           NVreg_RegistryDwordsPerDevice:charp
2023-01-11T21:20:39.5299932Z parm:           NVreg_RmMsg:charp
2023-01-11T21:20:39.5300223Z parm:           NVreg_GpuBlacklist:charp
2023-01-11T21:20:39.5300524Z parm:           NVreg_TemporaryFilePath:charp
2023-01-11T21:20:39.5300839Z parm:           NVreg_ExcludedGpus:charp
2023-01-11T21:20:39.5301145Z parm:           NVreg_DmaRemapPeerMmio:int
2023-01-11T21:20:39.5301431Z parm:           rm_firmware_active:charp
2023-01-11T21:20:39.5301690Z + set +e
2023-01-11T21:20:39.5301982Z + nvidia-smi
2023-01-11T21:20:39.5457775Z Wed Jan 11 21:20:39 2023       
2023-01-11T21:20:39.5458249Z +-----------------------------------------------------------------------------+
2023-01-11T21:20:39.5459120Z | NVIDIA-SMI 515.76       Driver Version: 515.76       CUDA Version: 11.7     |
2023-01-11T21:20:39.5460060Z |-------------------------------+----------------------+----------------------+
2023-01-11T21:20:39.5460565Z | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
2023-01-11T21:20:39.5461063Z | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
2023-01-11T21:20:39.5461419Z |                               |                      |               MIG M. |
2023-01-11T21:20:39.5461697Z |===============================+======================+======================|
2023-01-11T21:20:39.5507934Z |   0  Tesla M60           Off  | 00000000:00:1D.0 Off |           5106988244 |
2023-01-11T21:20:39.5508508Z | N/A   27C    P0    40W / 150W |      0MiB /  7680MiB |      0%      Default |
2023-01-11T21:20:39.5508857Z |                               |                      |                  N/A |
2023-01-11T21:20:39.5509307Z +-------------------------------+----------------------+----------------------+
2023-01-11T21:20:39.5555770Z |   1  Tesla M60           Off  | 00000000:00:1E.0 Off |           8030816435 |
2023-01-11T21:20:39.5556238Z | N/A   33C    P0    39W / 150W |      0MiB /  7680MiB |     66%      Default |
2023-01-11T21:20:39.5557122Z |                               |                      |                  N/A |
2023-01-11T21:20:39.5557770Z +-------------------------------+----------------------+----------------------+
2023-01-11T21:20:39.5558144Z                                                                                
2023-01-11T21:20:39.5558572Z +-----------------------------------------------------------------------------+
2023-01-11T21:20:39.5558926Z | Processes:                                                                  |
2023-01-11T21:20:39.5559278Z |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
2023-01-11T21:20:39.5559613Z |        ID   ID                                                   Usage      |
2023-01-11T21:20:39.5559914Z |=============================================================================|
2023-01-11T21:20:39.5560941Z |  No running processes found                                                 |
2023-01-11T21:20:39.5561967Z +-----------------------------------------------------------------------------+
2023-01-11T21:20:39.6116296Z + NVIDIA_SMI_STATUS=0
2023-01-11T21:20:39.6116875Z + '[' 0 -eq 0 ']'
2023-01-11T21:20:39.6117559Z + echo 'INFO: Ignoring allowed status 0'
2023-01-11T21:20:39.6118139Z + set -e
2023-01-11T21:20:39.6118457Z INFO: Ignoring allowed status 0
2023-01-11T21:20:39.6124590Z == Installing nvidia container toolkit for amzn2 ==
2023-01-11T21:20:39.6128216Z + sudo yum install -y yum-utils
2023-01-11T21:20:40.1594814Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2023-01-11T21:20:41.5984475Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version
2023-01-11T21:20:41.5985055Z Nothing to do
2023-01-11T21:20:41.6826124Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2023-01-11T21:20:42.2505372Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2023-01-11T21:20:42.2848102Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2023-01-11T21:20:42.2849361Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo
2023-01-11T21:20:42.2849897Z repo saved to /etc/yum.repos.d/nvidia-docker.repo
2023-01-11T21:20:42.2993370Z + sudo yum install -y nvidia-docker2
2023-01-11T21:20:42.8494988Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2023-01-11T21:20:44.1651835Z Package nvidia-docker2-2.11.0-1.noarch already installed and latest version
2023-01-11T21:20:44.1652583Z Nothing to do
2023-01-11T21:20:44.2454388Z + sudo systemctl restart docker
2023-01-11T21:21:04.3640479Z Command completed after 1 attempt(s).
2023-01-11T21:21:04.3693028Z ##[group]Run python3 -m pip install psutil==5.9.1
2023-01-11T21:21:04.3693423Z python3 -m pip install psutil==5.9.1
2023-01-11T21:21:04.3693732Z python3 -m pip install pynvml==11.4.1
2023-01-11T21:21:04.3694084Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2023-01-11T21:21:04.3694464Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}"
2023-01-11T21:21:04.3707805Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:21:04.3708106Z env:
2023-01-11T21:21:04.3708349Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:21:04.3708598Z   GPU_FLAG: --gpus all
2023-01-11T21:21:04.3708849Z ##[endgroup]
2023-01-11T21:21:04.6677449Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T21:21:04.6902508Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1)
2023-01-11T21:21:05.2607032Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T21:21:05.2829763Z Requirement already satisfied: pynvml==11.4.1 in /home/ec2-user/.local/lib/python3.7/site-packages (11.4.1)
2023-01-11T21:21:05.5654629Z Prepare all required actions
2023-01-11T21:21:05.5655013Z Getting action download info
2023-01-11T21:21:05.7710929Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:4a8bfae15cc25cc0785c1603ee87a9da8fd442ea)
2023-01-11T21:21:06.0261328Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a)
2023-01-11T21:21:06.1627184Z ##[group]Run ./.github/actions/download-build-artifacts
2023-01-11T21:21:06.1627491Z with:
2023-01-11T21:21:06.1627770Z   name: linux-bionic-cuda11.7-py3.10-gcc7
2023-01-11T21:21:06.1628031Z env:
2023-01-11T21:21:06.1628268Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:21:06.1628535Z   GPU_FLAG: --gpus all
2023-01-11T21:21:06.1628773Z ##[endgroup]
2023-01-11T21:21:06.1659310Z ##[group]Run seemethere/download-artifact-s3@v4
2023-01-11T21:21:06.1659596Z with:
2023-01-11T21:21:06.1659853Z   name: linux-bionic-cuda11.7-py3.10-gcc7
2023-01-11T21:21:06.1660158Z   s3-bucket: gha-artifacts
2023-01-11T21:21:06.1660478Z   region: us-east-1
2023-01-11T21:21:06.1660690Z env:
2023-01-11T21:21:06.1660925Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:21:06.1661184Z   GPU_FLAG: --gpus all
2023-01-11T21:21:06.1661409Z ##[endgroup]
2023-01-11T21:21:06.7443212Z Found 1 objects with prefix pytorch/pytorch/3896346758/linux-bionic-cuda11.7-py3.10-gcc7/
2023-01-11T21:21:06.7443812Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2023-01-11T21:21:13.5516330Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2023-01-11T21:21:13.5516666Z 
2023-01-11T21:21:13.5537617Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2023-01-11T21:21:13.5548115Z Artifact download has finished successfully
2023-01-11T21:21:13.5698883Z ##[group]Run unzip -o artifacts.zip
2023-01-11T21:21:13.5699214Z unzip -o artifacts.zip
2023-01-11T21:21:13.5713178Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:21:13.5713465Z env:
2023-01-11T21:21:13.5713712Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:21:13.5713984Z   GPU_FLAG: --gpus all
2023-01-11T21:21:13.5714217Z ##[endgroup]
2023-01-11T21:21:13.5757639Z Archive:  artifacts.zip
2023-01-11T21:21:13.5759682Z    creating: dist/
2023-01-11T21:21:15.6650632Z   inflating: dist/torch-2.0.0a0+git8419ddd-cp310-cp310-linux_x86_64.whl  
2023-01-11T21:21:15.6651080Z    creating: build/custom_test_artifacts/
2023-01-11T21:21:15.6651505Z    creating: build/custom_test_artifacts/custom-op-build/
2023-01-11T21:21:15.6652210Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2023-01-11T21:21:15.6658386Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log  
2023-01-11T21:21:15.6658936Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/
2023-01-11T21:21:15.6659509Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2023-01-11T21:21:15.6660047Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/
2023-01-11T21:21:15.6660605Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2023-01-11T21:21:15.6663016Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2023-01-11T21:21:15.6664159Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2023-01-11T21:21:15.6664705Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/
2023-01-11T21:21:15.6665280Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2023-01-11T21:21:15.6668203Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2023-01-11T21:21:15.6669470Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2023-01-11T21:21:15.6671271Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2023-01-11T21:21:15.6672242Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2023-01-11T21:21:15.6674417Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2023-01-11T21:21:15.6675540Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2023-01-11T21:21:15.6676165Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2023-01-11T21:21:15.6676756Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2023-01-11T21:21:15.6731182Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2023-01-11T21:21:15.6731918Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2023-01-11T21:21:15.6732637Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2023-01-11T21:21:15.6733385Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2023-01-11T21:21:15.6734109Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2023-01-11T21:21:15.6734808Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2023-01-11T21:21:15.6735497Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2023-01-11T21:21:15.6736204Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2023-01-11T21:21:15.6736909Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2023-01-11T21:21:15.6779663Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2023-01-11T21:21:15.6821160Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2023-01-11T21:21:15.6822164Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2023-01-11T21:21:15.6823042Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2023-01-11T21:21:15.6823867Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2023-01-11T21:21:15.6824759Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2023-01-11T21:21:15.6825571Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2023-01-11T21:21:15.6826504Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2023-01-11T21:21:15.6828503Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2023-01-11T21:21:15.6902305Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2023-01-11T21:21:15.6975835Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2023-01-11T21:21:15.6976488Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2023-01-11T21:21:15.6977044Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2023-01-11T21:21:15.6977866Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log  
2023-01-11T21:21:15.6978449Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2023-01-11T21:21:15.6979037Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2023-01-11T21:21:15.6979620Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2023-01-11T21:21:15.6980219Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2023-01-11T21:21:15.6980822Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2023-01-11T21:21:15.6981635Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2023-01-11T21:21:15.6982231Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2023-01-11T21:21:15.6983179Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2023-01-11T21:21:15.6983974Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2023-01-11T21:21:15.6984577Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2023-01-11T21:21:15.6985165Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2023-01-11T21:21:15.7007008Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2023-01-11T21:21:15.7125094Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2023-01-11T21:21:15.7125676Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2023-01-11T21:21:15.7126279Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2023-01-11T21:21:15.7126899Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2023-01-11T21:21:15.7127513Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2023-01-11T21:21:15.7128109Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2023-01-11T21:21:15.7128719Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2023-01-11T21:21:15.7129614Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2023-01-11T21:21:15.7130373Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2023-01-11T21:21:15.7131144Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2023-01-11T21:21:15.7131751Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2023-01-11T21:21:15.7152904Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2023-01-11T21:21:15.7240152Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2023-01-11T21:21:15.7240793Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2023-01-11T21:21:15.7241407Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2023-01-11T21:21:15.7241973Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2023-01-11T21:21:15.7242790Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2023-01-11T21:21:15.7244095Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2023-01-11T21:21:15.7245299Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2023-01-11T21:21:15.7248616Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2023-01-11T21:21:15.7249302Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2023-01-11T21:21:15.7250031Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2023-01-11T21:21:15.7345355Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2023-01-11T21:21:15.7410665Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2023-01-11T21:21:15.7411119Z    creating: build/custom_test_artifacts/jit-hook-build/
2023-01-11T21:21:15.7411579Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2023-01-11T21:21:15.7418352Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log  
2023-01-11T21:21:15.7418893Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/
2023-01-11T21:21:15.7419416Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2023-01-11T21:21:15.7419974Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/
2023-01-11T21:21:15.7420523Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2023-01-11T21:21:15.7422769Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2023-01-11T21:21:15.7423895Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2023-01-11T21:21:15.7424448Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/
2023-01-11T21:21:15.7425007Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2023-01-11T21:21:15.7427809Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2023-01-11T21:21:15.7428832Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2023-01-11T21:21:15.7430680Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2023-01-11T21:21:15.7431302Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2023-01-11T21:21:15.7433255Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2023-01-11T21:21:15.7434256Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2023-01-11T21:21:15.7434836Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2023-01-11T21:21:15.7435395Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2023-01-11T21:21:15.7489942Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2023-01-11T21:21:15.7490644Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2023-01-11T21:21:15.7491367Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2023-01-11T21:21:15.7492098Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2023-01-11T21:21:15.7492816Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2023-01-11T21:21:15.7493516Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2023-01-11T21:21:15.7494182Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2023-01-11T21:21:15.7494869Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2023-01-11T21:21:15.7495918Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2023-01-11T21:21:15.7538189Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2023-01-11T21:21:15.7579549Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2023-01-11T21:21:15.7580759Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2023-01-11T21:21:15.7581595Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2023-01-11T21:21:15.7582489Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2023-01-11T21:21:15.7583348Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2023-01-11T21:21:15.7584160Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2023-01-11T21:21:15.7585248Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2023-01-11T21:21:15.7587307Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2023-01-11T21:21:15.7660961Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2023-01-11T21:21:15.7735045Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2023-01-11T21:21:15.7735661Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2023-01-11T21:21:15.7736220Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2023-01-11T21:21:15.7736930Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log  
2023-01-11T21:21:15.7737482Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2023-01-11T21:21:15.7738008Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2023-01-11T21:21:15.7738595Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2023-01-11T21:21:15.7739373Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2023-01-11T21:21:15.7739981Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2023-01-11T21:21:15.7740695Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2023-01-11T21:21:15.7741393Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2023-01-11T21:21:15.7742715Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2023-01-11T21:21:15.7743395Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2023-01-11T21:21:15.7744155Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2023-01-11T21:21:15.7744773Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2023-01-11T21:21:15.7766247Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2023-01-11T21:21:15.7833795Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2023-01-11T21:21:15.7834432Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2023-01-11T21:21:15.7835048Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2023-01-11T21:21:15.7835603Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2023-01-11T21:21:15.7836471Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2023-01-11T21:21:15.7837524Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2023-01-11T21:21:15.7838052Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2023-01-11T21:21:15.7841323Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2023-01-11T21:21:15.7841994Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2023-01-11T21:21:15.7842916Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2023-01-11T21:21:15.7894507Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2023-01-11T21:21:15.7894994Z    creating: build/custom_test_artifacts/custom-backend-build/
2023-01-11T21:21:15.7895497Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2023-01-11T21:21:15.7902156Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log  
2023-01-11T21:21:15.7902727Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/
2023-01-11T21:21:15.7903300Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2023-01-11T21:21:15.7903888Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/
2023-01-11T21:21:15.7904446Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2023-01-11T21:21:15.7906750Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2023-01-11T21:21:15.7907875Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2023-01-11T21:21:15.7908478Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/
2023-01-11T21:21:15.7909041Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2023-01-11T21:21:15.7911712Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2023-01-11T21:21:15.7913176Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2023-01-11T21:21:15.7914911Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2023-01-11T21:21:15.7915541Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2023-01-11T21:21:15.7917312Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2023-01-11T21:21:15.7918229Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2023-01-11T21:21:15.7918966Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2023-01-11T21:21:15.7919539Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2023-01-11T21:21:15.7974045Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2023-01-11T21:21:15.7974796Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2023-01-11T21:21:15.7975543Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2023-01-11T21:21:15.7976304Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2023-01-11T21:21:15.7977028Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2023-01-11T21:21:15.7977754Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2023-01-11T21:21:15.7978628Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2023-01-11T21:21:15.7979372Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2023-01-11T21:21:15.7980075Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2023-01-11T21:21:15.8022286Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2023-01-11T21:21:15.8064431Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2023-01-11T21:21:15.8065552Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2023-01-11T21:21:15.8066383Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2023-01-11T21:21:15.8067196Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2023-01-11T21:21:15.8068047Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2023-01-11T21:21:15.8068965Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2023-01-11T21:21:15.8070230Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2023-01-11T21:21:15.8072307Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2023-01-11T21:21:15.8145967Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2023-01-11T21:21:15.8219578Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2023-01-11T21:21:15.8220249Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2023-01-11T21:21:15.8220832Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2023-01-11T21:21:15.8221544Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log  
2023-01-11T21:21:15.8222116Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2023-01-11T21:21:15.8222684Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2023-01-11T21:21:15.8223299Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2023-01-11T21:21:15.8224092Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2023-01-11T21:21:15.8224723Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2023-01-11T21:21:15.8225613Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2023-01-11T21:21:15.8226257Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2023-01-11T21:21:15.8227057Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2023-01-11T21:21:15.8227700Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2023-01-11T21:21:15.8228311Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2023-01-11T21:21:15.8228941Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2023-01-11T21:21:15.8234182Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2023-01-11T21:21:15.8388244Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2023-01-11T21:21:15.8388905Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2023-01-11T21:21:15.8389522Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2023-01-11T21:21:15.8390189Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2023-01-11T21:21:15.8390836Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2023-01-11T21:21:15.8391464Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2023-01-11T21:21:15.8392600Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2023-01-11T21:21:15.8393272Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2023-01-11T21:21:15.8393921Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2023-01-11T21:21:15.8394567Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2023-01-11T21:21:15.8395189Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2023-01-11T21:21:15.8416253Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2023-01-11T21:21:15.8478659Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2023-01-11T21:21:15.8479361Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2023-01-11T21:21:15.8479984Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2023-01-11T21:21:15.8480578Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2023-01-11T21:21:15.8481325Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2023-01-11T21:21:15.8482571Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2023-01-11T21:21:15.8483116Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2023-01-11T21:21:15.8486141Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2023-01-11T21:21:15.8486791Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2023-01-11T21:21:15.8487662Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2023-01-11T21:21:15.8611369Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2023-01-11T21:21:15.8659094Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2023-01-11T21:21:15.8659463Z    creating: build/lib/
2023-01-11T21:21:15.8660133Z   inflating: build/lib/libclog.a     
2023-01-11T21:21:15.8730347Z   inflating: build/lib/libgtest.a    
2023-01-11T21:21:15.8740505Z   inflating: build/lib/libpthreadpool.a  
2023-01-11T21:21:15.8837566Z   inflating: build/lib/libbenchmark.a  
2023-01-11T21:21:15.8846823Z   inflating: build/lib/libittnotify.a  
2023-01-11T21:21:15.8953421Z   inflating: build/lib/libprotobuf-lite.a  
2023-01-11T21:21:15.8985173Z   inflating: build/lib/libtensorpipe_uv.a  
2023-01-11T21:21:15.9062324Z   inflating: build/lib/libasmjit.a   
2023-01-11T21:21:15.9590340Z   inflating: build/lib/libprotobuf.a  
2023-01-11T21:21:15.9729424Z   inflating: build/lib/libgloo.a     
2023-01-11T21:21:15.9761762Z   inflating: build/lib/libfmt.a      
2023-01-11T21:21:15.9762292Z   inflating: build/lib/libfoxi_loader.a  
2023-01-11T21:21:15.9764683Z   inflating: build/lib/libcaffe2_nvrtc.so  
2023-01-11T21:21:15.9846427Z   inflating: build/lib/libc10.so     
2023-01-11T21:21:15.9847656Z   inflating: build/lib/libtorch_global_deps.so  
2023-01-11T21:21:16.0418057Z   inflating: build/lib/libprotoc.a   
2023-01-11T21:21:16.0427686Z   inflating: build/lib/libcpuinfo.a  
2023-01-11T21:21:16.0430281Z   inflating: build/lib/libnnpack_reference_layers.a  
2023-01-11T21:21:16.0439485Z   inflating: build/lib/libcpuinfo_internals.a  
2023-01-11T21:21:16.0457145Z   inflating: build/lib/libgmock.a    
2023-01-11T21:21:16.0457701Z   inflating: build/lib/libgtest_main.a  
2023-01-11T21:21:16.0458786Z   inflating: build/lib/libbenchmark_main.a  
2023-01-11T21:21:16.1116853Z   inflating: build/lib/libtensorpipe.a  
2023-01-11T21:21:17.0930313Z   inflating: build/lib/libdnnl.a     
2023-01-11T21:21:17.1072027Z   inflating: build/lib/libXNNPACK.a  
2023-01-11T21:21:17.1127091Z   inflating: build/lib/libc10_cuda.so  
2023-01-11T21:21:17.1142989Z   inflating: build/lib/libqnnpack.a  
2023-01-11T21:21:17.1143679Z   inflating: build/lib/libgmock_main.a  
2023-01-11T21:21:17.2691755Z   inflating: build/lib/libfbgemm.a   
2023-01-11T21:21:17.2715081Z   inflating: build/lib/libpytorch_qnnpack.a  
2023-01-11T21:21:17.3875286Z   inflating: build/lib/libdnnl_graph.a  
2023-01-11T21:21:17.4393173Z   inflating: build/lib/libkineto.a   
2023-01-11T21:21:17.4683907Z   inflating: build/lib/libtensorpipe_cuda.a  
2023-01-11T21:21:17.4729533Z   inflating: build/lib/libcaffe2_protos.a  
2023-01-11T21:21:17.4751751Z   inflating: build/lib/libnnpack.a   
2023-01-11T21:21:17.4800666Z   inflating: build/lib/libonnx_proto.a  
2023-01-11T21:21:17.5481566Z   inflating: build/lib/libonnx.a     
2023-01-11T21:21:17.5915499Z   inflating: build/lib/libgloo_cuda.a  
2023-01-11T21:21:19.9815566Z   inflating: build/lib/libtorch_cpu.so  
2023-01-11T21:21:19.9826196Z   inflating: build/lib/libunbox_lib.a  
2023-01-11T21:21:22.1166267Z   inflating: build/lib/libtorch_cuda.so  
2023-01-11T21:21:22.1167062Z   inflating: build/lib/libtorch.so   
2023-01-11T21:21:22.1170151Z   inflating: build/lib/libc10d_cuda_test.so  
2023-01-11T21:21:23.1092420Z   inflating: build/lib/libtorch_cuda_linalg.so  
2023-01-11T21:21:23.1153773Z   inflating: build/lib/libtorchbind_test.so  
2023-01-11T21:21:23.1177985Z   inflating: build/lib/libjitbackend_test.so  
2023-01-11T21:21:23.1209290Z   inflating: build/lib/libbackend_with_compiler.so  
2023-01-11T21:21:23.1214289Z   inflating: build/lib/libshm.so     
2023-01-11T21:21:23.3075109Z   inflating: build/lib/libtorch_python.so  
2023-01-11T21:21:23.3115406Z   inflating: build/lib/libnnapi_backend.so  
2023-01-11T21:21:23.3115717Z    creating: build/bin/
2023-01-11T21:21:23.3170301Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2023-01-11T21:21:23.3228009Z   inflating: build/bin/c10_DeviceGuard_test  
2023-01-11T21:21:23.3284587Z   inflating: build/bin/c10_Device_test  
2023-01-11T21:21:23.3349330Z   inflating: build/bin/c10_DispatchKeySet_test  
2023-01-11T21:21:23.3402976Z   inflating: build/bin/c10_StreamGuard_test  
2023-01-11T21:21:23.3457771Z   inflating: build/bin/c10_SymInt_test  
2023-01-11T21:21:23.3519806Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2023-01-11T21:21:23.3581668Z   inflating: build/bin/c10_InlineStreamGuard_test  
2023-01-11T21:21:23.3645087Z   inflating: build/bin/c10_SizesAndStrides_test  
2023-01-11T21:21:23.3698317Z   inflating: build/bin/c10_Array_test  
2023-01-11T21:21:23.3757696Z   inflating: build/bin/c10_Bitset_test  
2023-01-11T21:21:23.3836927Z   inflating: build/bin/c10_C++17_test  
2023-01-11T21:21:23.3890039Z   inflating: build/bin/c10_ConstexprCrc_test  
2023-01-11T21:21:23.3944451Z   inflating: build/bin/c10_DeadlockDetection_test  
2023-01-11T21:21:23.3999626Z   inflating: build/bin/c10_Half_test  
2023-01-11T21:21:23.4062371Z   inflating: build/bin/c10_LeftRight_test  
2023-01-11T21:21:23.4131683Z   inflating: build/bin/c10_Metaprogramming_test  
2023-01-11T21:21:23.4187235Z   inflating: build/bin/c10_Synchronized_test  
2023-01-11T21:21:23.4347997Z   inflating: build/bin/c10_SmallVectorTest  
2023-01-11T21:21:23.4411284Z   inflating: build/bin/c10_ThreadLocal_test  
2023-01-11T21:21:23.4469708Z   inflating: build/bin/c10_TypeIndex_test  
2023-01-11T21:21:23.4525806Z   inflating: build/bin/c10_TypeList_test  
2023-01-11T21:21:23.4579045Z   inflating: build/bin/c10_TypeTraits_test  
2023-01-11T21:21:23.4637306Z   inflating: build/bin/c10_accumulate_test  
2023-01-11T21:21:23.4699460Z   inflating: build/bin/c10_bfloat16_test  
2023-01-11T21:21:23.4761036Z   inflating: build/bin/c10_complex_math_test  
2023-01-11T21:21:23.4822123Z   inflating: build/bin/c10_complex_test  
2023-01-11T21:21:23.4942046Z   inflating: build/bin/c10_either_test  
2023-01-11T21:21:23.5000981Z   inflating: build/bin/c10_exception_test  
2023-01-11T21:21:23.5056402Z   inflating: build/bin/c10_flags_test  
2023-01-11T21:21:23.5241451Z   inflating: build/bin/c10_intrusive_ptr_test  
2023-01-11T21:21:23.5297724Z   inflating: build/bin/c10_irange_test  
2023-01-11T21:21:23.5360938Z   inflating: build/bin/c10_logging_test  
2023-01-11T21:21:23.5442492Z   inflating: build/bin/c10_optional_test  
2023-01-11T21:21:23.5510828Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2023-01-11T21:21:23.5571865Z   inflating: build/bin/c10_registry_test  
2023-01-11T21:21:23.5636327Z   inflating: build/bin/c10_string_view_test  
2023-01-11T21:21:23.5693568Z   inflating: build/bin/c10_tempfile_test  
2023-01-11T21:21:23.5755586Z   inflating: build/bin/c10_typeid_test  
2023-01-11T21:21:23.5816664Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2023-01-11T21:21:23.6338628Z   inflating: build/bin/protoc-3.13.0.0  
2023-01-11T21:21:23.6860587Z   inflating: build/bin/protoc        
2023-01-11T21:21:23.6920028Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream  
2023-01-11T21:21:23.6978783Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test  
2023-01-11T21:21:23.7037636Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device  
2023-01-11T21:21:23.7095620Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes  
2023-01-11T21:21:23.7154704Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads  
2023-01-11T21:21:23.7213701Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks  
2023-01-11T21:21:23.7267434Z   inflating: build/bin/c10_cuda_CUDATest  
2023-01-11T21:21:23.7326406Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block  
2023-01-11T21:21:23.7650590Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2023-01-11T21:21:23.8011854Z   inflating: build/bin/vec_test_all_types_AVX2  
2023-01-11T21:21:23.8071687Z   inflating: build/bin/HashStoreTest  
2023-01-11T21:21:23.8138696Z   inflating: build/bin/TCPStoreTest  
2023-01-11T21:21:23.8198535Z   inflating: build/bin/FileStoreTest  
2023-01-11T21:21:23.8214707Z   inflating: build/bin/ProcessGroupMPITest  
2023-01-11T21:21:23.8277905Z   inflating: build/bin/test_edge_op_registration  
2023-01-11T21:21:23.8281318Z   inflating: build/bin/example_allreduce  
2023-01-11T21:21:23.8340009Z   inflating: build/bin/Dimname_test  
2023-01-11T21:21:23.8420495Z   inflating: build/bin/Dict_test     
2023-01-11T21:21:23.8491739Z   inflating: build/bin/MaybeOwned_test  
2023-01-11T21:21:23.8554820Z   inflating: build/bin/NamedTensor_test  
2023-01-11T21:21:23.8620712Z   inflating: build/bin/apply_utils_test  
2023-01-11T21:21:23.8688008Z   inflating: build/bin/basic         
2023-01-11T21:21:23.8753095Z   inflating: build/bin/atest         
2023-01-11T21:21:23.8812998Z   inflating: build/bin/broadcast_test  
2023-01-11T21:21:23.8877161Z   inflating: build/bin/cpu_generator_test  
2023-01-11T21:21:23.8935410Z   inflating: build/bin/cpu_profiling_allocator_test  
2023-01-11T21:21:23.9033632Z   inflating: build/bin/cpu_rng_test  
2023-01-11T21:21:23.9090073Z   inflating: build/bin/dispatch_key_set_test  
2023-01-11T21:21:23.9146038Z   inflating: build/bin/dlconvertor_test  
2023-01-11T21:21:23.9211935Z   inflating: build/bin/extension_backend_test  
2023-01-11T21:21:23.9274197Z   inflating: build/bin/half_test     
2023-01-11T21:21:23.9328758Z   inflating: build/bin/lazy_tensor_test  
2023-01-11T21:21:23.9389050Z   inflating: build/bin/math_kernel_test  
2023-01-11T21:21:23.9493669Z   inflating: build/bin/ivalue_test   
2023-01-11T21:21:23.9553794Z   inflating: build/bin/memory_format_test  
2023-01-11T21:21:23.9613817Z   inflating: build/bin/memory_overlapping_test  
2023-01-11T21:21:23.9670166Z   inflating: build/bin/operator_name_test  
2023-01-11T21:21:23.9728720Z   inflating: build/bin/mobile_memory_cleanup  
2023-01-11T21:21:23.9790773Z   inflating: build/bin/native_test   
2023-01-11T21:21:23.9846535Z   inflating: build/bin/operators_test  
2023-01-11T21:21:23.9905204Z   inflating: build/bin/packedtensoraccessor_test  
2023-01-11T21:21:23.9969146Z   inflating: build/bin/quantized_test  
2023-01-11T21:21:24.0042244Z   inflating: build/bin/pow_test      
2023-01-11T21:21:24.0096760Z   inflating: build/bin/reduce_ops_test  
2023-01-11T21:21:24.0153416Z   inflating: build/bin/reportMemoryUsage_test  
2023-01-11T21:21:24.0216042Z   inflating: build/bin/scalar_tensor_test  
2023-01-11T21:21:24.0279587Z   inflating: build/bin/scalar_test   
2023-01-11T21:21:24.0337794Z   inflating: build/bin/stride_properties_test  
2023-01-11T21:21:24.0424450Z   inflating: build/bin/tensor_iterator_test  
2023-01-11T21:21:24.0485498Z   inflating: build/bin/type_ptr_test  
2023-01-11T21:21:24.0488866Z   inflating: build/bin/thread_init_test  
2023-01-11T21:21:24.0550952Z   inflating: build/bin/test_parallel  
2023-01-11T21:21:24.0606605Z   inflating: build/bin/variant_test  
2023-01-11T21:21:24.0665275Z   inflating: build/bin/undefined_tensor_test  
2023-01-11T21:21:24.0732581Z   inflating: build/bin/type_test     
2023-01-11T21:21:24.0734010Z   inflating: build/bin/verify_api_visibility  
2023-01-11T21:21:24.0810451Z   inflating: build/bin/legacy_vmap_test  
2023-01-11T21:21:24.0867085Z   inflating: build/bin/weakref_test  
2023-01-11T21:21:24.0923860Z   inflating: build/bin/wrapdim_test  
2023-01-11T21:21:24.1043856Z   inflating: build/bin/List_test     
2023-01-11T21:21:24.1110076Z   inflating: build/bin/IListRef_test  
2023-01-11T21:21:24.1164233Z   inflating: build/bin/xla_tensor_test  
2023-01-11T21:21:24.1298040Z   inflating: build/bin/kernel_function_legacy_test  
2023-01-11T21:21:24.1369871Z   inflating: build/bin/KernelFunction_test  
2023-01-11T21:21:24.1475781Z   inflating: build/bin/kernel_function_test  
2023-01-11T21:21:24.1616675Z   inflating: build/bin/kernel_lambda_legacy_test  
2023-01-11T21:21:24.1683476Z   inflating: build/bin/kernel_stackbased_test  
2023-01-11T21:21:24.1797270Z   inflating: build/bin/kernel_lambda_test  
2023-01-11T21:21:24.1853885Z   inflating: build/bin/CppSignature_test  
2023-01-11T21:21:24.1959038Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2023-01-11T21:21:24.2012195Z   inflating: build/bin/op_allowlist_test  
2023-01-11T21:21:24.2072245Z   inflating: build/bin/inline_container_test  
2023-01-11T21:21:24.2135213Z   inflating: build/bin/backend_fallback_test  
2023-01-11T21:21:24.2452286Z   inflating: build/bin/op_registration_test  
2023-01-11T21:21:24.2510627Z   inflating: build/bin/cuda_apply_test  
2023-01-11T21:21:24.2588909Z   inflating: build/bin/cuda_complex_math_test  
2023-01-11T21:21:24.2643567Z   inflating: build/bin/cuda_device_test  
2023-01-11T21:21:24.2703276Z   inflating: build/bin/cuda_caching_host_allocator_test  
2023-01-11T21:21:24.2769713Z   inflating: build/bin/cuda_atomic_ops_test  
2023-01-11T21:21:24.2825120Z   inflating: build/bin/cuda_dlconvertor_test  
2023-01-11T21:21:24.2889753Z   inflating: build/bin/cuda_complex_test  
2023-01-11T21:21:24.2955343Z   inflating: build/bin/cuda_cub_test  
2023-01-11T21:21:24.3011187Z   inflating: build/bin/cuda_integer_divider_test  
2023-01-11T21:21:24.3085504Z   inflating: build/bin/cuda_distributions_test  
2023-01-11T21:21:24.3144164Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2023-01-11T21:21:24.3211272Z   inflating: build/bin/cuda_stream_test  
2023-01-11T21:21:24.3276733Z   inflating: build/bin/cuda_generator_test  
2023-01-11T21:21:24.3330593Z   inflating: build/bin/cuda_optional_test  
2023-01-11T21:21:24.3385169Z   inflating: build/bin/cuda_half_test  
2023-01-11T21:21:24.3442221Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2023-01-11T21:21:24.3459974Z   inflating: build/bin/tutorial_tensorexpr  
2023-01-11T21:21:24.3532227Z   inflating: build/bin/ProcessGroupGlooTest  
2023-01-11T21:21:24.3585941Z   inflating: build/bin/cuda_cudnn_test  
2023-01-11T21:21:24.3644989Z   inflating: build/bin/ProcessGroupUCCTest  
2023-01-11T21:21:24.3704828Z   inflating: build/bin/test_dist_autograd  
2023-01-11T21:21:24.3769438Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2023-01-11T21:21:24.3833817Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2023-01-11T21:21:24.3901708Z   inflating: build/bin/ProcessGroupNCCLTest  
2023-01-11T21:21:24.3978737Z   inflating: build/bin/test_cpp_rpc  
2023-01-11T21:21:24.3981499Z   inflating: build/bin/parallel_benchmark  
2023-01-11T21:21:24.4057229Z   inflating: build/bin/test_mobile_nnc  
2023-01-11T21:21:24.4068693Z   inflating: build/bin/aot_model_compiler_test  
2023-01-11T21:21:24.4126948Z   inflating: build/bin/cuda_vectorized_test  
2023-01-11T21:21:24.4132840Z   inflating: build/bin/torch_shm_manager  
2023-01-11T21:21:24.4522463Z   inflating: build/bin/test_lazy     
2023-01-11T21:21:24.5432542Z   inflating: build/bin/test_tensorexpr  
2023-01-11T21:21:24.6753610Z   inflating: build/bin/test_api      
2023-01-11T21:21:24.7957887Z   inflating: build/bin/test_jit      
2023-01-11T21:21:24.7961026Z   inflating: .pytorch-test-times.json  
2023-01-11T21:21:24.7990868Z ##[group]Run df -H
2023-01-11T21:21:24.7991125Z df -H
2023-01-11T21:21:24.8004920Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:21:24.8005199Z env:
2023-01-11T21:21:24.8005442Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:21:24.8005710Z   GPU_FLAG: --gpus all
2023-01-11T21:21:24.8005941Z ##[endgroup]
2023-01-11T21:21:24.8045460Z Filesystem      Size  Used Avail Use% Mounted on
2023-01-11T21:21:24.8045796Z devtmpfs        129G     0  129G   0% /dev
2023-01-11T21:21:24.8046066Z tmpfs           129G   25k  129G   1% /dev/shm
2023-01-11T21:21:24.8046341Z tmpfs           129G  553k  129G   1% /run
2023-01-11T21:21:24.8046633Z tmpfs           129G     0  129G   0% /sys/fs/cgroup
2023-01-11T21:21:24.8047164Z /dev/xvda1      162G   27G  135G  17% /
2023-01-11T21:21:24.8070170Z ##[group]Run .github/scripts/parse_ref.py
2023-01-11T21:21:24.8070536Z .github/scripts/parse_ref.py
2023-01-11T21:21:24.8083023Z shell: /usr/bin/bash -e {0}
2023-01-11T21:21:24.8083271Z env:
2023-01-11T21:21:24.8083508Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:21:24.8083861Z   GPU_FLAG: --gpus all
2023-01-11T21:21:24.8084108Z ##[endgroup]
2023-01-11T21:21:24.8373683Z ##[group]Run set -x
2023-01-11T21:21:24.8374064Z set -x
2023-01-11T21:21:24.8374294Z 
2023-01-11T21:21:24.8374547Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2023-01-11T21:21:24.8374899Z   TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh
2023-01-11T21:21:24.8375249Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2023-01-11T21:21:24.8375573Z   TEST_COMMAND=.jenkins/onnx/test.sh
2023-01-11T21:21:24.8375826Z else
2023-01-11T21:21:24.8376105Z   TEST_COMMAND=.jenkins/pytorch/test.sh
2023-01-11T21:21:24.8376372Z fi
2023-01-11T21:21:24.8376573Z 
2023-01-11T21:21:24.8376896Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}")
2023-01-11T21:21:24.8377213Z 
2023-01-11T21:21:24.8377492Z # sanitize the input commit message and PR body here:
2023-01-11T21:21:24.8377791Z #
2023-01-11T21:21:24.8378174Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment
2023-01-11T21:21:24.8378670Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028
2023-01-11T21:21:24.8379099Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}"
2023-01-11T21:21:24.8379479Z PR_BODY="${PR_BODY//[$'\n\r']}"
2023-01-11T21:21:24.8379735Z 
2023-01-11T21:21:24.8380077Z # then trim all special characters like single and double quotes to avoid unescaped inputs to
2023-01-11T21:21:24.8380454Z # wreak havoc internally
2023-01-11T21:21:24.8380777Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}"
2023-01-11T21:21:24.8381086Z export PR_BODY="${PR_BODY//[\'\"]}"
2023-01-11T21:21:24.8381347Z 
2023-01-11T21:21:24.8381659Z # detached container should get cleaned up by teardown_ec2_linux
2023-01-11T21:21:24.8382054Z # TODO: Stop building test binaries as part of the build phase
2023-01-11T21:21:24.8382432Z # Used for GPU_FLAG since that doesn't play nice
2023-01-11T21:21:24.8382762Z # shellcheck disable=SC2086,SC2090
2023-01-11T21:21:24.8383061Z container_name=$(docker run \
2023-01-11T21:21:24.8383321Z   ${GPU_FLAG:-} \
2023-01-11T21:21:24.8383592Z   -e BUILD_ENVIRONMENT \
2023-01-11T21:21:24.8383869Z   -e PR_NUMBER \
2023-01-11T21:21:24.8384118Z   -e GITHUB_ACTIONS \
2023-01-11T21:21:24.8384377Z   -e BASE_SHA \
2023-01-11T21:21:24.8384627Z   -e BRANCH \
2023-01-11T21:21:24.8384851Z   -e SHA1 \
2023-01-11T21:21:24.8385106Z   -e AWS_DEFAULT_REGION \
2023-01-11T21:21:24.8385377Z   -e IN_WHEEL_TEST \
2023-01-11T21:21:24.8385618Z   -e SHARD_NUMBER \
2023-01-11T21:21:24.8385882Z   -e TEST_CONFIG \
2023-01-11T21:21:24.8386147Z   -e NUM_TEST_SHARDS \
2023-01-11T21:21:24.8386414Z   -e PR_BODY \
2023-01-11T21:21:24.8386662Z   -e COMMIT_MESSAGES \
2023-01-11T21:21:24.8386943Z   -e CONTINUE_THROUGH_ERROR \
2023-01-11T21:21:24.8387243Z   -e PYTORCH_RETRY_TEST_CASES \
2023-01-11T21:21:24.8387538Z   -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \
2023-01-11T21:21:24.8387826Z   -e PR_LABELS \
2023-01-11T21:21:24.8388119Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2023-01-11T21:21:24.8388393Z   -e SCCACHE_BUCKET \
2023-01-11T21:21:24.8388671Z   -e SCCACHE_S3_KEY_PREFIX \
2023-01-11T21:21:24.8388938Z   -e XLA_CUDA \
2023-01-11T21:21:24.8389205Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2023-01-11T21:21:24.8389531Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2023-01-11T21:21:24.8389863Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2023-01-11T21:21:24.8390214Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2023-01-11T21:21:24.8390520Z   --ulimit stack=10485760:83886080 \
2023-01-11T21:21:24.8391005Z   --security-opt seccomp=unconfined \
2023-01-11T21:21:24.8391318Z   --cap-add=SYS_PTRACE \
2023-01-11T21:21:24.8391568Z   --ipc=host \
2023-01-11T21:21:24.8392103Z   --shm-size="${SHM_SIZE}" \
2023-01-11T21:21:24.8392388Z   --tty \
2023-01-11T21:21:24.8392614Z   --detach \
2023-01-11T21:21:24.8392884Z   --name="${container_name}" \
2023-01-11T21:21:24.8393161Z   --user jenkins \
2023-01-11T21:21:24.8393471Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2023-01-11T21:21:24.8393812Z   -w /var/lib/jenkins/workspace \
2023-01-11T21:21:24.8394090Z   "${DOCKER_IMAGE}"
2023-01-11T21:21:24.8394330Z )
2023-01-11T21:21:24.8394619Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}"
2023-01-11T21:21:24.8395068Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2023-01-11T21:21:24.8406694Z shell: /usr/bin/bash -e {0}
2023-01-11T21:21:24.8406932Z env:
2023-01-11T21:21:24.8407174Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:21:24.8407440Z   GPU_FLAG: --gpus all
2023-01-11T21:21:24.8407753Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.7-py3.10-gcc7
2023-01-11T21:21:24.8408066Z   PR_NUMBER: 
2023-01-11T21:21:24.8408291Z   BRANCH: 
2023-01-11T21:21:24.8408550Z   SHA1: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:21:24.8408893Z   BASE_SHA: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:21:24.8409204Z   PYTORCH_RETRY_TEST_CASES: 1
2023-01-11T21:21:24.8409474Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2023-01-11T21:21:24.8409752Z   TEST_CONFIG: distributed
2023-01-11T21:21:24.8410004Z   SHARD_NUMBER: 2
2023-01-11T21:21:24.8410226Z   NUM_TEST_SHARDS: 3
2023-01-11T21:21:24.8410462Z   PR_BODY: 
2023-01-11T21:21:24.8410716Z   CONTINUE_THROUGH_ERROR: False
2023-01-11T21:21:24.8411052Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2023-01-11T21:21:24.8411366Z   SCCACHE_S3_KEY_PREFIX: trunk
2023-01-11T21:21:24.8411624Z   SHM_SIZE: 2g
2023-01-11T21:21:24.8412124Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T21:21:24.8412584Z   XLA_CUDA: 
2023-01-11T21:21:24.8412934Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2023-01-11T21:21:24.8413313Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2023-01-11T21:21:24.8413595Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2023-01-11T21:21:24.8413867Z ##[endgroup]
2023-01-11T21:21:24.8442128Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2023-01-11T21:21:24.8442609Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *onnx* ]]
2023-01-11T21:21:24.8442949Z + TEST_COMMAND=.jenkins/pytorch/test.sh
2023-01-11T21:21:24.8446115Z ++ git cherry -v origin/master
2023-01-11T21:21:24.9005553Z + COMMIT_MESSAGES='+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into '\''input'\''
2023-01-11T21:21:24.9006055Z + 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch
2023-01-11T21:21:24.9006621Z + 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e '\''other'\'' instead of '\''output'\'' in documentation'
2023-01-11T21:21:24.9007840Z + COMMIT_MESSAGES='+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into '\''input'\''+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e '\''other'\'' instead of '\''output'\'' in documentation'
2023-01-11T21:21:24.9008437Z + PR_BODY=
2023-01-11T21:21:24.9009840Z + export 'COMMIT_MESSAGES=+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into input+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e other instead of output in documentation'
2023-01-11T21:21:24.9011128Z + COMMIT_MESSAGES='+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into input+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e other instead of output in documentation'
2023-01-11T21:21:24.9011755Z + export PR_BODY=
2023-01-11T21:21:24.9011971Z + PR_BODY=
2023-01-11T21:21:24.9019347Z +++ nproc --ignore=2
2023-01-11T21:21:24.9031696Z ++ docker run --gpus all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e CONTINUE_THROUGH_ERROR -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS --env-file=/tmp/github_env_3896346758 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T21:21:39.0962703Z + container_name=c5cb3f34fdb745cbab81ead9bbd8979c6cd118fca3d50c4cb582659c15cdaef3
2023-01-11T21:21:39.0963891Z + echo DOCKER_CONTAINER_ID=c5cb3f34fdb745cbab81ead9bbd8979c6cd118fca3d50c4cb582659c15cdaef3
2023-01-11T21:21:39.0968325Z ++ echo dist/torch-2.0.0a0+git8419ddd-cp310-cp310-linux_x86_64.whl
2023-01-11T21:21:39.0971176Z + docker exec -t c5cb3f34fdb745cbab81ead9bbd8979c6cd118fca3d50c4cb582659c15cdaef3 sh -c 'pip install dist/torch-2.0.0a0+git8419ddd-cp310-cp310-linux_x86_64.whl[opt-einsum] && .jenkins/pytorch/test.sh'
2023-01-11T21:21:39.6479537Z Processing ./dist/torch-2.0.0a0+git8419ddd-cp310-cp310-linux_x86_64.whl
2023-01-11T21:21:40.6067083Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch==2.0.0a0+git8419ddd) (4.4.0)
2023-01-11T21:21:40.6069971Z Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torch==2.0.0a0+git8419ddd) (1.11.1)
2023-01-11T21:21:40.6074169Z Requirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from torch==2.0.0a0+git8419ddd) (2.6.3)
2023-01-11T21:21:40.6090581Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/lib/python3.10/site-packages (from torch==2.0.0a0+git8419ddd) (3.3.0)
2023-01-11T21:21:40.6169511Z Requirement already satisfied: numpy>=1.7 in /opt/conda/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.0.0a0+git8419ddd) (1.21.2)
2023-01-11T21:21:40.6382236Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torch==2.0.0a0+git8419ddd) (1.2.1)
2023-01-11T21:21:41.5400409Z Installing collected packages: torch
2023-01-11T21:21:51.1002122Z Successfully installed torch-2.0.0a0+git8419ddd
2023-01-11T21:21:51.2729858Z + echo 'Environment variables:'
2023-01-11T21:21:51.2730911Z Environment variables:
2023-01-11T21:21:51.2733164Z + env
2023-01-11T21:21:51.2737170Z SHARD_NUMBER=2
2023-01-11T21:21:51.2737643Z NV_LIBCUBLAS_DEV_VERSION=11.10.1.25-1
2023-01-11T21:21:51.2738056Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-7
2023-01-11T21:21:51.2738509Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2023-01-11T21:21:51.2739095Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.13.4-1+cuda11.7
2023-01-11T21:21:51.2739467Z UCC_HOME=/usr
2023-01-11T21:21:51.2740079Z BUILD_ENVIRONMENT=linux-bionic-cuda11.7-py3.10-gcc7
2023-01-11T21:21:51.2740516Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2023-01-11T21:21:51.2741028Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-7=11.7.3.21-1
2023-01-11T21:21:51.2741343Z INSTALLED_DB=yes
2023-01-11T21:21:51.2741653Z HOSTNAME=c5cb3f34fdb7
2023-01-11T21:21:51.2741993Z GITHUB_REF_NAME=ciflow/trunk/91627
2023-01-11T21:21:51.2742335Z GITHUB_API_URL=https://api.github.com
2023-01-11T21:21:51.2743066Z GITHUB_REPOSITORY_OWNER_ID=21003710
2023-01-11T21:21:51.2743936Z OPENSSL_DIR=/opt/openssl
2023-01-11T21:21:51.2744515Z UCC_COMMIT=1c7a7127186e7836f73aafbd7697bbc274a77eee
2023-01-11T21:21:51.2745957Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_9859b153-ea70-43de-a5f9-b04ff3eb5e69
2023-01-11T21:21:51.2746654Z CUDA_PATH=/usr/local/cuda
2023-01-11T21:21:51.2747525Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2023-01-11T21:21:51.2752649Z GITHUB_RUN_ATTEMPT=1
2023-01-11T21:21:51.2753142Z TEST_CONFIG=distributed
2023-01-11T21:21:51.2753769Z NV_LIBNPP_VERSION=11.7.3.21-1
2023-01-11T21:21:51.2754316Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-7=11.7.50-1
2023-01-11T21:21:51.2754675Z GITHUB_REPOSITORY_OWNER=pytorch
2023-01-11T21:21:51.2755019Z GITHUB_ACTIONS=true
2023-01-11T21:21:51.2755499Z NVIDIA_VISIBLE_DEVICES=all
2023-01-11T21:21:51.2755891Z NV_NVPROF_VERSION=11.7.50-1
2023-01-11T21:21:51.2756234Z NV_LIBCUSPARSE_VERSION=11.7.3.50-1
2023-01-11T21:21:51.2756707Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/trunk.yml@refs/tags/ciflow/trunk/91627
2023-01-11T21:21:51.2757190Z NVIDIA_PRODUCT_NAME=CUDA
2023-01-11T21:21:51.2757456Z CI=true
2023-01-11T21:21:51.2802905Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2023-01-11T21:21:51.2803359Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-7=11.10.1.25-1
2023-01-11T21:21:51.2803670Z BRANCH=
2023-01-11T21:21:51.2803881Z GITHUB_HEAD_REF=
2023-01-11T21:21:51.2804176Z UCX_COMMIT=31e74cac7bee0ef66bef2af72e7d86d9c282e5ab
2023-01-11T21:21:51.2804534Z GITHUB_ACTOR=pytorch-bot[bot]
2023-01-11T21:21:51.2804854Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2023-01-11T21:21:51.2805145Z GITHUB_ACTION_REF=
2023-01-11T21:21:51.2805389Z NCCL_VERSION=2.13.4-1
2023-01-11T21:21:51.2805621Z GITHUB_ACTION=__self
2023-01-11T21:21:51.2805860Z GITHUB_REF_PROTECTED=false
2023-01-11T21:21:51.2806284Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2023-01-11T21:21:51.2806651Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2023-01-11T21:21:51.2807249Z ***
2023-01-11T21:21:51.2807468Z INSTALLED_VISION=yes
2023-01-11T21:21:51.2807709Z NVARCH=x86_64
2023-01-11T21:21:51.2808003Z NV_LIBCUSPARSE_DEV_VERSION=11.7.3.50-1
2023-01-11T21:21:51.2808255Z HOME=/var/lib/jenkins
2023-01-11T21:21:51.2808782Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_9859b153-ea70-43de-a5f9-b04ff3eb5e69
2023-01-11T21:21:51.2809189Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2023-01-11T21:21:51.2809460Z NVIDIA_CUDA_END_OF_LIFE=1
2023-01-11T21:21:51.2809711Z GITHUB_ACTION_REPOSITORY=
2023-01-11T21:21:51.2809969Z GITHUB_REF_TYPE=tag
2023-01-11T21:21:51.2810288Z NV_LIBNCCL_PACKAGE_VERSION=2.13.4-1
2023-01-11T21:21:51.2810574Z GITHUB_RETENTION_DAYS=90
2023-01-11T21:21:51.2810985Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2023-01-11T21:21:51.2811422Z NV_LIBNCCL_PACKAGE=libnccl2=2.13.4-1+cuda11.7
2023-01-11T21:21:51.2811987Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_9859b153-ea70-43de-a5f9-b04ff3eb5e69
2023-01-11T21:21:51.2812428Z DEBIAN_FRONTEND=noninteractive
2023-01-11T21:21:51.2812804Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2023-01-11T21:21:51.2813113Z GITHUB_REF=refs/tags/ciflow/trunk/91627
2023-01-11T21:21:51.2813446Z NV_CUDA_LIB_VERSION=11.7.0-1
2023-01-11T21:21:51.2813779Z GITHUB_SHA=8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:21:51.2814079Z INSTALLED_PROTOBUF=yes
2023-01-11T21:21:51.2814365Z GITHUB_REPOSITORY_ID=65600975
2023-01-11T21:21:51.2814647Z GITHUB_RUN_ID=3896346758
2023-01-11T21:21:51.2814996Z NV_LIBNPP_PACKAGE=libnpp-11-7=11.7.3.21-1
2023-01-11T21:21:51.2815318Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2023-01-11T21:21:51.2815630Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2023-01-11T21:21:51.2815955Z NV_NVTX_VERSION=11.7.50-1
2023-01-11T21:21:51.2816220Z CONTINUE_THROUGH_ERROR=False
2023-01-11T21:21:51.2816541Z GITHUB_SERVER_URL=https://github.com
2023-01-11T21:21:51.2816835Z MAX_JOBS=30
2023-01-11T21:21:51.2817264Z GITHUB_ACTOR_ID=54816060
2023-01-11T21:21:51.2817672Z NV_LIBCUBLAS_VERSION=11.10.1.25-1
2023-01-11T21:21:51.2818087Z NV_LIBCUBLAS_PACKAGE=libcublas-11-7=11.10.1.25-1
2023-01-11T21:21:51.2818590Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2023-01-11T21:21:51.2818959Z UCX_HOME=/usr
2023-01-11T21:21:51.2819224Z PYTORCH_RETRY_TEST_CASES=1
2023-01-11T21:21:51.2819555Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2023-01-11T21:21:51.2819930Z BASE_SHA=8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:21:51.2820294Z NV_CUDA_CUDART_DEV_VERSION=11.7.60-1
2023-01-11T21:21:51.2820553Z PR_BODY=
2023-01-11T21:21:51.2820793Z GITHUB_BASE_REF=
2023-01-11T21:21:51.2821038Z TERM=xterm
2023-01-11T21:21:51.2821252Z XLA_CUDA=
2023-01-11T21:21:51.2821535Z NV_NVML_DEV_VERSION=11.7.50-1
2023-01-11T21:21:51.2821825Z TORCH_CUDA_ARCH_LIST=Maxwell
2023-01-11T21:21:51.2822081Z CUDA_VERSION=11.7.0
2023-01-11T21:21:51.2822445Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-7
2023-01-11T21:21:51.2822770Z OPENSSL_ROOT_DIR=/opt/openssl
2023-01-11T21:21:51.2823335Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_9859b153-ea70-43de-a5f9-b04ff3eb5e69
2023-01-11T21:21:51.2823750Z GITHUB_JOB=test
2023-01-11T21:21:51.2824014Z SCCACHE_S3_KEY_PREFIX=trunk
2023-01-11T21:21:51.2824656Z COMMIT_MESSAGES=+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into input+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e other instead of output in documentation
2023-01-11T21:21:51.2825367Z NVIDIA_DRIVER_CAPABILITIES=compute,utility
2023-01-11T21:21:51.2825651Z NUM_TEST_SHARDS=3
2023-01-11T21:21:51.2825898Z PR_NUMBER=
2023-01-11T21:21:51.2826449Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_9859b153-ea70-43de-a5f9-b04ff3eb5e69
2023-01-11T21:21:51.2826833Z SHLVL=1
2023-01-11T21:21:51.2827196Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-7
2023-01-11T21:21:51.2827549Z GITHUB_REPOSITORY=pytorch/pytorch
2023-01-11T21:21:51.2828839Z NVIDIA_REQUIRE_CUDA=cuda>=11.7 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=510,driver<511 brand=unknown,driver>=510,driver<511 brand=nvidia,driver>=510,driver<511 brand=nvidiartx,driver>=510,driver<511 brand=quadro,driver>=510,driver<511 brand=quadrortx,driver>=510,driver<511 brand=titan,driver>=510,driver<511 brand=titanrtx,driver>=510,driver<511 brand=geforce,driver>=510,driver<511 brand=geforcertx,driver>=510,driver<511
2023-01-11T21:21:51.2830059Z NV_LIBNPP_DEV_VERSION=11.7.3.21-1
2023-01-11T21:21:51.2830389Z SHA1=8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:21:51.2830684Z GITHUB_EVENT_NAME=push
2023-01-11T21:21:51.2831004Z NV_CUDA_CUDART_VERSION=11.7.60-1
2023-01-11T21:21:51.2831376Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2023-01-11T21:21:51.2831663Z GITHUB_RUN_NUMBER=22986
2023-01-11T21:21:51.2832300Z GITHUB_WORKFLOW=trunk
2023-01-11T21:21:51.2832750Z PATH=/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-01-11T21:21:51.2833221Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.13.4-1
2023-01-11T21:21:51.2833589Z GITHUB_WORKFLOW_SHA=8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:21:51.2834088Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T21:21:51.2834517Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot]
2023-01-11T21:21:51.2834812Z _=/usr/bin/env
2023-01-11T21:21:51.2835213Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2023-01-11T21:21:51.2961424Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.10/site-packages/torch
2023-01-11T21:21:51.2962110Z + TORCH_BIN_DIR=/opt/conda/lib/python3.10/site-packages/torch/bin
2023-01-11T21:21:51.2962597Z + TORCH_LIB_DIR=/opt/conda/lib/python3.10/site-packages/torch/lib
2023-01-11T21:21:51.2963064Z + TORCH_TEST_DIR=/opt/conda/lib/python3.10/site-packages/torch/test
2023-01-11T21:21:51.2963362Z + BUILD_DIR=build
2023-01-11T21:21:51.2963628Z + BUILD_RENAMED_DIR=build_renamed
2023-01-11T21:21:51.2963902Z + BUILD_BIN_DIR=build/bin
2023-01-11T21:21:51.2964147Z + export VALGRIND=ON
2023-01-11T21:21:51.2964399Z + VALGRIND=ON
2023-01-11T21:21:51.2964668Z + export TORCH_INDUCTOR_INSTALL_GXX=ON
2023-01-11T21:21:51.2964948Z + TORCH_INDUCTOR_INSTALL_GXX=ON
2023-01-11T21:21:51.2965362Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *clang9* ]]
2023-01-11T21:21:51.2965802Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 != *bazel* ]]
2023-01-11T21:21:51.2968321Z ++ realpath build/custom_test_artifacts
2023-01-11T21:21:51.2976678Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2023-01-11T21:21:51.2979986Z ++ dirname .jenkins/pytorch/test.sh
2023-01-11T21:21:51.2986854Z + source .jenkins/pytorch/common.sh
2023-01-11T21:21:51.2990724Z +++ dirname .jenkins/pytorch/common.sh
2023-01-11T21:21:51.3002468Z ++ source .jenkins/pytorch/common_utils.sh
2023-01-11T21:21:51.3004416Z +++ declare -f -t trap_add
2023-01-11T21:21:51.3010821Z ++ set -ex
2023-01-11T21:21:51.3011282Z ++ [[ linux-bionic-cuda11.7-py3.10-gcc7 == *rocm* ]]
2023-01-11T21:21:51.3011853Z ++ BUILD_TEST_LIBTORCH=0
2023-01-11T21:21:51.3012494Z + echo 'Environment variables'
2023-01-11T21:21:51.3012973Z Environment variables
2023-01-11T21:21:51.3013403Z + env
2023-01-11T21:21:51.3021180Z SHARD_NUMBER=2
2023-01-11T21:21:51.3021790Z NV_LIBCUBLAS_DEV_VERSION=11.10.1.25-1
2023-01-11T21:21:51.3022477Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-7
2023-01-11T21:21:51.3023189Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2023-01-11T21:21:51.3023989Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.13.4-1+cuda11.7
2023-01-11T21:21:51.3024351Z UCC_HOME=/usr
2023-01-11T21:21:51.3024731Z BUILD_ENVIRONMENT=linux-bionic-cuda11.7-py3.10-gcc7
2023-01-11T21:21:51.3025079Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2023-01-11T21:21:51.3025449Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-7=11.7.3.21-1
2023-01-11T21:21:51.3025747Z INSTALLED_DB=yes
2023-01-11T21:21:51.3025997Z HOSTNAME=c5cb3f34fdb7
2023-01-11T21:21:51.3026257Z GITHUB_REF_NAME=ciflow/trunk/91627
2023-01-11T21:21:51.3026578Z GITHUB_API_URL=https://api.github.com
2023-01-11T21:21:51.3026893Z GITHUB_REPOSITORY_OWNER_ID=21003710
2023-01-11T21:21:51.3027180Z OPENSSL_DIR=/opt/openssl
2023-01-11T21:21:51.3027466Z UCC_COMMIT=1c7a7127186e7836f73aafbd7697bbc274a77eee
2023-01-11T21:21:51.3028070Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_9859b153-ea70-43de-a5f9-b04ff3eb5e69
2023-01-11T21:21:51.3028484Z CUDA_PATH=/usr/local/cuda
2023-01-11T21:21:51.3028957Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2023-01-11T21:21:51.3029339Z GITHUB_RUN_ATTEMPT=1
2023-01-11T21:21:51.3029603Z TEST_CONFIG=distributed
2023-01-11T21:21:51.3029883Z NV_LIBNPP_VERSION=11.7.3.21-1
2023-01-11T21:21:51.3030258Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-7=11.7.50-1
2023-01-11T21:21:51.3030580Z GITHUB_REPOSITORY_OWNER=pytorch
2023-01-11T21:21:51.3030833Z GITHUB_ACTIONS=true
2023-01-11T21:21:51.3031093Z NVIDIA_VISIBLE_DEVICES=all
2023-01-11T21:21:51.3031387Z NV_NVPROF_VERSION=11.7.50-1
2023-01-11T21:21:51.3031672Z NV_LIBCUSPARSE_VERSION=11.7.3.50-1
2023-01-11T21:21:51.3032681Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/trunk.yml@refs/tags/ciflow/trunk/91627
2023-01-11T21:21:51.3033055Z NVIDIA_PRODUCT_NAME=CUDA
2023-01-11T21:21:51.3033295Z CI=true
2023-01-11T21:21:51.3033525Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2023-01-11T21:21:51.3033945Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-7=11.10.1.25-1
2023-01-11T21:21:51.3034239Z BRANCH=
2023-01-11T21:21:51.3034470Z GITHUB_HEAD_REF=
2023-01-11T21:21:51.3034760Z UCX_COMMIT=31e74cac7bee0ef66bef2af72e7d86d9c282e5ab
2023-01-11T21:21:51.3035368Z GITHUB_ACTOR=pytorch-bot[bot]
2023-01-11T21:21:51.3035699Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2023-01-11T21:21:51.3035973Z GITHUB_ACTION_REF=
2023-01-11T21:21:51.3036249Z NCCL_VERSION=2.13.4-1
2023-01-11T21:21:51.3036501Z GITHUB_ACTION=__self
2023-01-11T21:21:51.3036727Z VALGRIND=ON
2023-01-11T21:21:51.3036976Z GITHUB_REF_PROTECTED=false
2023-01-11T21:21:51.3037429Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2023-01-11T21:21:51.3037795Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2023-01-11T21:21:51.3038144Z ***
2023-01-11T21:21:51.3038375Z INSTALLED_VISION=yes
2023-01-11T21:21:51.3038603Z NVARCH=x86_64
2023-01-11T21:21:51.3038903Z NV_LIBCUSPARSE_DEV_VERSION=11.7.3.50-1
2023-01-11T21:21:51.3039185Z HOME=/var/lib/jenkins
2023-01-11T21:21:51.3039697Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_9859b153-ea70-43de-a5f9-b04ff3eb5e69
2023-01-11T21:21:51.3040108Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2023-01-11T21:21:51.3040389Z NVIDIA_CUDA_END_OF_LIFE=1
2023-01-11T21:21:51.3040661Z GITHUB_ACTION_REPOSITORY=
2023-01-11T21:21:51.3040906Z GITHUB_REF_TYPE=tag
2023-01-11T21:21:51.3041205Z NV_LIBNCCL_PACKAGE_VERSION=2.13.4-1
2023-01-11T21:21:51.3041486Z GITHUB_RETENTION_DAYS=90
2023-01-11T21:21:51.3041840Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2023-01-11T21:21:51.3042249Z NV_LIBNCCL_PACKAGE=libnccl2=2.13.4-1+cuda11.7
2023-01-11T21:21:51.3042801Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_9859b153-ea70-43de-a5f9-b04ff3eb5e69
2023-01-11T21:21:51.3043188Z DEBIAN_FRONTEND=noninteractive
2023-01-11T21:21:51.3043536Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2023-01-11T21:21:51.3043846Z GITHUB_REF=refs/tags/ciflow/trunk/91627
2023-01-11T21:21:51.3044143Z NV_CUDA_LIB_VERSION=11.7.0-1
2023-01-11T21:21:51.3044460Z GITHUB_SHA=8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:21:51.3044763Z INSTALLED_PROTOBUF=yes
2023-01-11T21:21:51.3045017Z GITHUB_REPOSITORY_ID=65600975
2023-01-11T21:21:51.3045292Z GITHUB_RUN_ID=3896346758
2023-01-11T21:21:51.3045632Z NV_LIBNPP_PACKAGE=libnpp-11-7=11.7.3.21-1
2023-01-11T21:21:51.3045915Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2023-01-11T21:21:51.3046206Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2023-01-11T21:21:51.3046516Z NV_NVTX_VERSION=11.7.50-1
2023-01-11T21:21:51.3046766Z CONTINUE_THROUGH_ERROR=False
2023-01-11T21:21:51.3047067Z GITHUB_SERVER_URL=https://github.com
2023-01-11T21:21:51.3047330Z MAX_JOBS=30
2023-01-11T21:21:51.3047572Z GITHUB_ACTOR_ID=54816060
2023-01-11T21:21:51.3047853Z NV_LIBCUBLAS_VERSION=11.10.1.25-1
2023-01-11T21:21:51.3048226Z NV_LIBCUBLAS_PACKAGE=libcublas-11-7=11.10.1.25-1
2023-01-11T21:21:51.3048711Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2023-01-11T21:21:51.3049041Z UCX_HOME=/usr
2023-01-11T21:21:51.3049295Z PYTORCH_RETRY_TEST_CASES=1
2023-01-11T21:21:51.3049626Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2023-01-11T21:21:51.3049968Z BASE_SHA=8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:21:51.3050322Z NV_CUDA_CUDART_DEV_VERSION=11.7.60-1
2023-01-11T21:21:51.3050584Z PR_BODY=
2023-01-11T21:21:51.3050797Z GITHUB_BASE_REF=
2023-01-11T21:21:51.3051029Z TERM=xterm
2023-01-11T21:21:51.3051283Z TORCH_INDUCTOR_INSTALL_GXX=ON
2023-01-11T21:21:51.3051518Z XLA_CUDA=
2023-01-11T21:21:51.3051786Z NV_NVML_DEV_VERSION=11.7.50-1
2023-01-11T21:21:51.3052050Z TORCH_CUDA_ARCH_LIST=Maxwell
2023-01-11T21:21:51.3052291Z CUDA_VERSION=11.7.0
2023-01-11T21:21:51.3052634Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-7
2023-01-11T21:21:51.3052932Z OPENSSL_ROOT_DIR=/opt/openssl
2023-01-11T21:21:51.3053454Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_9859b153-ea70-43de-a5f9-b04ff3eb5e69
2023-01-11T21:21:51.3053842Z GITHUB_JOB=test
2023-01-11T21:21:51.3054095Z SCCACHE_S3_KEY_PREFIX=trunk
2023-01-11T21:21:51.3054787Z COMMIT_MESSAGES=+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into input+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e other instead of output in documentation
2023-01-11T21:21:51.3055446Z NVIDIA_DRIVER_CAPABILITIES=compute,utility
2023-01-11T21:21:51.3055726Z NUM_TEST_SHARDS=3
2023-01-11T21:21:51.3055961Z PR_NUMBER=
2023-01-11T21:21:51.3056470Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_9859b153-ea70-43de-a5f9-b04ff3eb5e69
2023-01-11T21:21:51.3056854Z SHLVL=1
2023-01-11T21:21:51.3057197Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-7
2023-01-11T21:21:51.3057498Z GITHUB_REPOSITORY=pytorch/pytorch
2023-01-11T21:21:51.3058672Z NVIDIA_REQUIRE_CUDA=cuda>=11.7 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=510,driver<511 brand=unknown,driver>=510,driver<511 brand=nvidia,driver>=510,driver<511 brand=nvidiartx,driver>=510,driver<511 brand=quadro,driver>=510,driver<511 brand=quadrortx,driver>=510,driver<511 brand=titan,driver>=510,driver<511 brand=titanrtx,driver>=510,driver<511 brand=geforce,driver>=510,driver<511 brand=geforcertx,driver>=510,driver<511
2023-01-11T21:21:51.3059793Z NV_LIBNPP_DEV_VERSION=11.7.3.21-1
2023-01-11T21:21:51.3060112Z SHA1=8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:21:51.3060408Z GITHUB_EVENT_NAME=push
2023-01-11T21:21:51.3060726Z NV_CUDA_CUDART_VERSION=11.7.60-1
2023-01-11T21:21:51.3061077Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2023-01-11T21:21:51.3061363Z GITHUB_RUN_NUMBER=22986
2023-01-11T21:21:51.3061602Z GITHUB_WORKFLOW=trunk
2023-01-11T21:21:51.3062017Z PATH=/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-01-11T21:21:51.3062470Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.13.4-1
2023-01-11T21:21:51.3062815Z GITHUB_WORKFLOW_SHA=8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:21:51.3063268Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T21:21:51.3063687Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot]
2023-01-11T21:21:51.3063963Z _=/usr/bin/env
2023-01-11T21:21:51.3064229Z + echo 'Testing pytorch'
2023-01-11T21:21:51.3064481Z Testing pytorch
2023-01-11T21:21:51.3064747Z + export LANG=C.UTF-8
2023-01-11T21:21:51.3064987Z + LANG=C.UTF-8
2023-01-11T21:21:51.3065207Z + PR_NUMBER=
2023-01-11T21:21:51.3065455Z + [[ distributed == \d\e\f\a\u\l\t ]]
2023-01-11T21:21:51.3065729Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2023-01-11T21:21:51.3066114Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *rocm* ]]
2023-01-11T21:21:51.3066428Z + [[ distributed == \s\l\o\w ]]
2023-01-11T21:21:51.3066823Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *slow-gradcheck* ]]
2023-01-11T21:21:51.3067271Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *cuda* ]]
2023-01-11T21:21:51.3067620Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2023-01-11T21:21:51.3067910Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2023-01-11T21:21:51.3068200Z + [[ distributed == *crossref* ]]
2023-01-11T21:21:51.3068473Z + [[ distributed == *dynamo* ]]
2023-01-11T21:21:51.3068744Z + [[ distributed == *inductor* ]]
2023-01-11T21:21:51.3069111Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *rocm* ]]
2023-01-11T21:21:51.3069542Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 != *-bazel-* ]]
2023-01-11T21:21:51.3069924Z + pip_install --user ninja==1.10.2
2023-01-11T21:21:51.3070294Z + pip install --progress-bar off --user ninja==1.10.2
2023-01-11T21:21:51.8386820Z Collecting ninja==1.10.2
2023-01-11T21:21:51.8598067Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2023-01-11T21:21:52.7526085Z Installing collected packages: ninja
2023-01-11T21:21:52.7627295Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2023-01-11T21:21:52.7628453Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2023-01-11T21:21:52.7696269Z Successfully installed ninja-1.10.2
2023-01-11T21:21:52.8338407Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-01-11T21:21:52.8339680Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-01-11T21:21:52.8340665Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *asan* ]]
2023-01-11T21:21:52.8341115Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *-tsan* ]]
2023-01-11T21:21:52.8341476Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2023-01-11T21:21:52.8341802Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2023-01-11T21:21:52.8348442Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *tbb* ]]
2023-01-11T21:21:52.8363602Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *libtorch* ]]
2023-01-11T21:21:52.8364607Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *-bazel-* ]]
2023-01-11T21:21:52.8365109Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *-tsan* ]]
2023-01-11T21:21:52.8366341Z + cd test
2023-01-11T21:21:52.8367148Z + python -c 'import torch; print(torch.__config__.show())'
2023-01-11T21:21:54.4790392Z PyTorch built with:
2023-01-11T21:21:54.4791264Z   - GCC 7.5
2023-01-11T21:21:54.4792256Z   - C++ Version: 201703
2023-01-11T21:21:54.4792924Z   - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2023-01-11T21:21:54.4793481Z   - Intel(R) MKL-DNN v2.7.2 (Git Hash fbec3e25a559ee252022ae066817b204e106a6ba)
2023-01-11T21:21:54.4793885Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2023-01-11T21:21:54.4794260Z   - LAPACK is enabled (usually provided by MKL)
2023-01-11T21:21:54.4794568Z   - NNPACK is enabled
2023-01-11T21:21:54.4794912Z   - CPU capability usage: AVX2
2023-01-11T21:21:54.4795232Z   - CUDA Runtime 11.7
2023-01-11T21:21:54.4795610Z   - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52
2023-01-11T21:21:54.4795951Z   - CuDNN 8.5
2023-01-11T21:21:54.4796214Z   - Magma 2.6.1
2023-01-11T21:21:54.4799378Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.7, CUDNN_VERSION=8.5.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.0.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 
2023-01-11T21:21:54.4801723Z 
2023-01-11T21:21:54.7158358Z + cd test
2023-01-11T21:21:54.7159023Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2023-01-11T21:21:56.2874711Z ATen/Parallel:
2023-01-11T21:21:56.2893188Z 	at::get_num_threads() : 16
2023-01-11T21:21:56.2893522Z 	at::get_num_interop_threads() : 16
2023-01-11T21:21:56.2893823Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2023-01-11T21:21:56.2894364Z 	omp_get_max_threads() : 16
2023-01-11T21:21:56.2895116Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2023-01-11T21:21:56.2895521Z 	mkl_get_max_threads() : 16
2023-01-11T21:21:56.2895951Z Intel(R) MKL-DNN v2.7.2 (Git Hash fbec3e25a559ee252022ae066817b204e106a6ba)
2023-01-11T21:21:56.2896319Z std::thread::hardware_concurrency() : 32
2023-01-11T21:21:56.2896609Z Environment variables:
2023-01-11T21:21:56.2896860Z 	OMP_NUM_THREADS : [not set]
2023-01-11T21:21:56.2897145Z 	MKL_NUM_THREADS : [not set]
2023-01-11T21:21:56.2897423Z ATen parallel backend: OpenMP
2023-01-11T21:21:56.2897602Z 
2023-01-11T21:21:56.5248102Z + [[ distributed == *backward* ]]
2023-01-11T21:21:56.5248762Z + [[ distributed == *xla* ]]
2023-01-11T21:21:56.5249382Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2023-01-11T21:21:56.5249992Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *libtorch* ]]
2023-01-11T21:21:56.5250333Z + [[ distributed == distributed ]]
2023-01-11T21:21:56.5250622Z + install_filelock
2023-01-11T21:21:56.5250932Z + pip_install filelock
2023-01-11T21:21:56.5251280Z + pip install --progress-bar off filelock
2023-01-11T21:21:57.0359727Z Collecting filelock
2023-01-11T21:21:57.0551807Z   Downloading filelock-3.9.0-py3-none-any.whl (9.7 kB)
2023-01-11T21:21:57.9251349Z Installing collected packages: filelock
2023-01-11T21:21:57.9611292Z Successfully installed filelock-3.9.0
2023-01-11T21:21:58.0244525Z + install_triton
2023-01-11T21:21:58.0244800Z + local commit
2023-01-11T21:21:58.0245040Z + [[ distributed == *rocm* ]]
2023-01-11T21:21:58.0248953Z ++ get_pinned_commit triton
2023-01-11T21:21:58.0249667Z ++ cat .github/ci_commit_pins/triton.txt
2023-01-11T21:21:58.0263551Z + commit=0d7e7532279e45672555e344646f5c19c3972331
2023-01-11T21:21:58.0264918Z + pip_install --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2023-01-11T21:21:58.0265679Z + pip install --progress-bar off --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2023-01-11T21:21:58.4948587Z Collecting git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2023-01-11T21:21:58.4954523Z   Cloning https://github.com/openai/triton (to revision 0d7e7532279e45672555e344646f5c19c3972331) to /tmp/pip-req-build-bih1owh7
2023-01-11T21:21:58.4975064Z   Running command git clone --filter=blob:none --quiet https://github.com/openai/triton /tmp/pip-req-build-bih1owh7
2023-01-11T21:21:59.2726785Z   Running command git rev-parse -q --verify 'sha^0d7e7532279e45672555e344646f5c19c3972331'
2023-01-11T21:21:59.2747378Z   Running command git fetch -q https://github.com/openai/triton 0d7e7532279e45672555e344646f5c19c3972331
2023-01-11T21:21:59.6810227Z   Running command git checkout -q 0d7e7532279e45672555e344646f5c19c3972331
2023-01-11T21:22:00.1351669Z   Resolved https://github.com/openai/triton to commit 0d7e7532279e45672555e344646f5c19c3972331
2023-01-11T21:22:00.1353273Z   Running command git submodule update --init --recursive -q
2023-01-11T21:22:00.7504463Z   Preparing metadata (setup.py) ... [?25l- done
2023-01-11T21:22:00.9614835Z [?25hCollecting cmake
2023-01-11T21:22:00.9849913Z   Downloading cmake-3.25.0-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (23.7 MB)
2023-01-11T21:22:01.3572188Z Requirement already satisfied: filelock in /opt/conda/lib/python3.10/site-packages (from triton==2.0.0) (3.9.0)
2023-01-11T21:22:01.3575209Z Requirement already satisfied: torch in /opt/conda/lib/python3.10/site-packages (from triton==2.0.0) (2.0.0a0+git8419ddd)
2023-01-11T21:22:01.3828687Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch->triton==2.0.0) (4.4.0)
2023-01-11T21:22:01.3833223Z Requirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from torch->triton==2.0.0) (2.6.3)
2023-01-11T21:22:01.3838565Z Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torch->triton==2.0.0) (1.11.1)
2023-01-11T21:22:01.4049278Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torch->triton==2.0.0) (1.2.1)
2023-01-11T21:22:01.4119766Z Building wheels for collected packages: triton
2023-01-11T21:22:53.6976056Z   Building wheel for triton (setup.py) ... [?25l- \ | / - \ | / - \ done
2023-01-11T21:22:53.7456769Z [?25h  Created wheel for triton: filename=triton-2.0.0-cp310-cp310-linux_x86_64.whl size=15377935 sha256=fcb9c1ce070c3098e49f3a02ad3a3facc01644876c4b1fb906a9fd0820196a51
2023-01-11T21:22:53.7458804Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/3f/1d/23/1c2bc47d618a44f9c949aea4b7e355e737a1f1ed208f009295
2023-01-11T21:22:53.7478041Z Successfully built triton
2023-01-11T21:22:54.6284061Z Installing collected packages: cmake, triton
2023-01-11T21:22:57.5251556Z Successfully installed cmake-3.25.0 triton-2.0.0
2023-01-11T21:22:57.6280817Z + pip_install --user jinja2
2023-01-11T21:22:57.6281236Z + pip install --progress-bar off --user jinja2
2023-01-11T21:22:58.1778430Z Collecting jinja2
2023-01-11T21:22:58.1972536Z   Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
2023-01-11T21:22:58.2112039Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.10/site-packages (from jinja2) (2.1.1)
2023-01-11T21:22:59.1282973Z Installing collected packages: jinja2
2023-01-11T21:22:59.2316226Z Successfully installed jinja2-3.1.2
2023-01-11T21:22:59.2996199Z + test_distributed
2023-01-11T21:22:59.2996825Z + echo 'Testing distributed python tests'
2023-01-11T21:22:59.2997149Z Testing distributed python tests
2023-01-11T21:22:59.2999418Z + python test/run_test.py --distributed-tests --shard 2 3 --verbose
2023-01-11T21:23:01.5351825Z Ignoring disabled issues:  []
2023-01-11T21:23:01.5741035Z /var/lib/jenkins/workspace/test/run_test.py:1169: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2023-01-11T21:23:01.5741631Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2023-01-11T21:23:01.5747810Z Found test time stats from artifacts
2023-01-11T21:23:01.5766230Z Selected tests:
2023-01-11T21:23:01.5766861Z  distributed/algorithms/quantization/test_quantization
2023-01-11T21:23:01.5767419Z  distributed/test_distributed_spawn
2023-01-11T21:23:01.5767757Z  distributed/pipeline/sync/test_worker
2023-01-11T21:23:01.5768117Z  distributed/pipeline/sync/test_pipeline
2023-01-11T21:23:01.5768657Z  distributed/pipeline/sync/test_microbatch
2023-01-11T21:23:01.5769009Z  distributed/pipeline/sync/test_deferred_batch_norm
2023-01-11T21:23:01.5769341Z  distributed/pipeline/sync/test_bugs
2023-01-11T21:23:01.5769881Z  distributed/pipeline/sync/skip/test_tracker
2023-01-11T21:23:01.5770237Z  distributed/pipeline/sync/skip/test_leak
2023-01-11T21:23:01.5770558Z  distributed/pipeline/sync/skip/test_api
2023-01-11T21:23:01.5770957Z  distributed/fsdp/test_shard_utils
2023-01-11T21:23:01.5771475Z  distributed/checkpoint/test_2d_model_state_checkpoint
2023-01-11T21:23:01.5771858Z  distributed/_shard/sharded_tensor/ops/test_math_ops
2023-01-11T21:23:01.5772252Z  distributed/checkpoint/test_dedup_tensors
2023-01-11T21:23:01.5772767Z  distributed/checkpoint/test_nested_dict
2023-01-11T21:23:01.5773090Z  distributed/_composable/test_contract
2023-01-11T21:23:01.5773384Z  distributed/fsdp/test_utils
2023-01-11T21:23:01.5773814Z  distributed/fsdp/test_fsdp_fx
2023-01-11T21:23:01.5774208Z  distributed/elastic/utils/util_test
2023-01-11T21:23:01.5774538Z  distributed/rpc/test_share_memory
2023-01-11T21:23:01.5774809Z  distributed/test_data_parallel
2023-01-11T21:23:01.5775381Z  distributed/fsdp/test_fsdp_multiple_forward
2023-01-11T21:23:01.5775737Z  distributed/fsdp/test_fsdp_multiple_wrapping
2023-01-11T21:23:01.5776067Z  distributed/_shard/sharded_optim/test_sharded_optim
2023-01-11T21:23:01.5776664Z  distributed/_shard/sharded_tensor/test_sharded_tensor_reshard
2023-01-11T21:23:01.5777048Z  distributed/fsdp/test_distributed_checkpoint
2023-01-11T21:23:01.5777796Z  distributed/_shard/sharding_spec/test_sharding_spec
2023-01-11T21:23:01.5778328Z  distributed/_shard/sharded_tensor/ops/test_linear
2023-01-11T21:23:01.5778695Z  distributed/_shard/sharded_tensor/ops/test_elementwise_ops
2023-01-11T21:23:01.5779183Z  distributed/tensor/parallel/test_tp_examples
2023-01-11T21:23:01.5779647Z  distributed/_shard/test_partial_tensor
2023-01-11T21:23:01.5779988Z  distributed/_shard/sharding_plan/test_sharding_plan
2023-01-11T21:23:01.5780323Z  distributed/_composable/test_replicate
2023-01-11T21:23:01.5780881Z  distributed/fsdp/test_fsdp_tp_integration
2023-01-11T21:23:01.5781198Z  distributed/test_dynamo_distributed
2023-01-11T21:23:01.5781506Z  distributed/checkpoint/test_checkpoint
2023-01-11T21:23:01.5781962Z  distributed/elastic/multiprocessing/api_test
2023-01-11T21:23:01.5782385Z  distributed/_tensor/test_matrix_ops
2023-01-11T21:23:01.5782684Z  distributed/fsdp/test_fsdp_meta
2023-01-11T21:23:01.5782953Z  distributed/fsdp/test_fsdp_comm
2023-01-11T21:23:01.5783499Z  distributed/test_c10d_spawn_gloo
2023-01-11T21:23:01.5783797Z  distributed/test_c10d_common
2023-01-11T21:23:01.5784061Z  distributed/_tensor/test_dtensor
2023-01-11T21:23:01.5784382Z  distributed/_tensor/test_device_mesh
2023-01-11T21:23:01.5784903Z  distributed/test_pg_wrapper
2023-01-11T21:23:01.5785183Z  distributed/fsdp/test_wrap
2023-01-11T21:23:01.5785484Z  distributed/_shard/sharded_tensor/test_sharded_tensor
2023-01-11T21:23:01.5785934Z  distributed/fsdp/test_fsdp_optim_state
2023-01-11T21:23:01.5786411Z  distributed/fsdp/test_fsdp_state_dict
2023-01-11T21:23:01.5786684Z  distributed/test_c10d_nccl
2023-01-11T21:23:01.5939592Z Prioritized test from test file changes.
2023-01-11T21:23:01.5939990Z reordering tests for PR:
2023-01-11T21:23:01.5940783Z prioritized: ['distributed/checkpoint/test_2d_model_state_checkpoint', 'distributed/_tensor/test_device_mesh', 'distributed/fsdp/test_fsdp_optim_state']
2023-01-11T21:23:01.5946204Z the rest: ['distributed/algorithms/quantization/test_quantization', 'distributed/test_distributed_spawn', 'distributed/pipeline/sync/test_worker', 'distributed/pipeline/sync/test_pipeline', 'distributed/pipeline/sync/test_microbatch', 'distributed/pipeline/sync/test_deferred_batch_norm', 'distributed/pipeline/sync/test_bugs', 'distributed/pipeline/sync/skip/test_tracker', 'distributed/pipeline/sync/skip/test_leak', 'distributed/pipeline/sync/skip/test_api', 'distributed/fsdp/test_shard_utils', 'distributed/_shard/sharded_tensor/ops/test_math_ops', 'distributed/checkpoint/test_dedup_tensors', 'distributed/checkpoint/test_nested_dict', 'distributed/_composable/test_contract', 'distributed/fsdp/test_utils', 'distributed/fsdp/test_fsdp_fx', 'distributed/elastic/utils/util_test', 'distributed/rpc/test_share_memory', 'distributed/test_data_parallel', 'distributed/fsdp/test_fsdp_multiple_forward', 'distributed/fsdp/test_fsdp_multiple_wrapping', 'distributed/_shard/sharded_optim/test_sharded_optim', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard', 'distributed/fsdp/test_distributed_checkpoint', 'distributed/_shard/sharding_spec/test_sharding_spec', 'distributed/_shard/sharded_tensor/ops/test_linear', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops', 'distributed/tensor/parallel/test_tp_examples', 'distributed/_shard/test_partial_tensor', 'distributed/_shard/sharding_plan/test_sharding_plan', 'distributed/_composable/test_replicate', 'distributed/fsdp/test_fsdp_tp_integration', 'distributed/test_dynamo_distributed', 'distributed/checkpoint/test_checkpoint', 'distributed/elastic/multiprocessing/api_test', 'distributed/_tensor/test_matrix_ops', 'distributed/fsdp/test_fsdp_meta', 'distributed/fsdp/test_fsdp_comm', 'distributed/test_c10d_spawn_gloo', 'distributed/test_c10d_common', 'distributed/_tensor/test_dtensor', 'distributed/test_pg_wrapper', 'distributed/fsdp/test_wrap', 'distributed/_shard/sharded_tensor/test_sharded_tensor', 'distributed/fsdp/test_fsdp_state_dict', 'distributed/test_c10d_nccl']
2023-01-11T21:23:01.5949805Z 
2023-01-11T21:23:01.5950720Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2023-01-11T21:23:01.6157511Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2023-01-11T21:23:01.6328120Z parallel (file granularity) tests:
2023-01-11T21:23:01.6328747Z  
2023-01-11T21:23:01.6329240Z serial (file granularity) tests:
2023-01-11T21:23:01.6329905Z  distributed/checkpoint/test_2d_model_state_checkpoint
2023-01-11T21:23:01.6330548Z  distributed/_tensor/test_device_mesh
2023-01-11T21:23:01.6331097Z  distributed/fsdp/test_fsdp_optim_state
2023-01-11T21:23:01.6331721Z  distributed/algorithms/quantization/test_quantization
2023-01-11T21:23:01.6332229Z  distributed/test_distributed_spawn
2023-01-11T21:23:01.6332523Z  distributed/pipeline/sync/test_worker
2023-01-11T21:23:01.6332858Z  distributed/pipeline/sync/test_pipeline
2023-01-11T21:23:01.6333189Z  distributed/pipeline/sync/test_microbatch
2023-01-11T21:23:01.6333509Z  distributed/pipeline/sync/test_deferred_batch_norm
2023-01-11T21:23:01.6333837Z  distributed/pipeline/sync/test_bugs
2023-01-11T21:23:01.6334159Z  distributed/pipeline/sync/skip/test_tracker
2023-01-11T21:23:01.6334485Z  distributed/pipeline/sync/skip/test_leak
2023-01-11T21:23:01.6334785Z  distributed/pipeline/sync/skip/test_api
2023-01-11T21:23:01.6335089Z  distributed/fsdp/test_shard_utils
2023-01-11T21:23:01.6335415Z  distributed/_shard/sharded_tensor/ops/test_math_ops
2023-01-11T21:23:01.6335737Z  distributed/checkpoint/test_dedup_tensors
2023-01-11T21:23:01.6336056Z  distributed/checkpoint/test_nested_dict
2023-01-11T21:23:01.6336389Z  distributed/_composable/test_contract
2023-01-11T21:23:01.6336668Z  distributed/fsdp/test_utils
2023-01-11T21:23:01.6336948Z  distributed/fsdp/test_fsdp_fx
2023-01-11T21:23:01.6337242Z  distributed/elastic/utils/util_test
2023-01-11T21:23:01.6337543Z  distributed/rpc/test_share_memory
2023-01-11T21:23:01.6337816Z  distributed/test_data_parallel
2023-01-11T21:23:01.6338127Z  distributed/fsdp/test_fsdp_multiple_forward
2023-01-11T21:23:01.6338459Z  distributed/fsdp/test_fsdp_multiple_wrapping
2023-01-11T21:23:01.6338785Z  distributed/_shard/sharded_optim/test_sharded_optim
2023-01-11T21:23:01.6339152Z  distributed/_shard/sharded_tensor/test_sharded_tensor_reshard
2023-01-11T21:23:01.6339509Z  distributed/fsdp/test_distributed_checkpoint
2023-01-11T21:23:01.6339834Z  distributed/_shard/sharding_spec/test_sharding_spec
2023-01-11T21:23:01.6340178Z  distributed/_shard/sharded_tensor/ops/test_linear
2023-01-11T21:23:01.6340543Z  distributed/_shard/sharded_tensor/ops/test_elementwise_ops
2023-01-11T21:23:01.6340877Z  distributed/tensor/parallel/test_tp_examples
2023-01-11T21:23:01.6341194Z  distributed/_shard/test_partial_tensor
2023-01-11T21:23:01.6341524Z  distributed/_shard/sharding_plan/test_sharding_plan
2023-01-11T21:23:01.6341833Z  distributed/_composable/test_replicate
2023-01-11T21:23:01.6342163Z  distributed/fsdp/test_fsdp_tp_integration
2023-01-11T21:23:01.6342473Z  distributed/test_dynamo_distributed
2023-01-11T21:23:01.6342782Z  distributed/checkpoint/test_checkpoint
2023-01-11T21:23:01.6343093Z  distributed/elastic/multiprocessing/api_test
2023-01-11T21:23:01.6343412Z  distributed/_tensor/test_matrix_ops
2023-01-11T21:23:01.6343707Z  distributed/fsdp/test_fsdp_meta
2023-01-11T21:23:01.6343978Z  distributed/fsdp/test_fsdp_comm
2023-01-11T21:23:01.6344262Z  distributed/test_c10d_spawn_gloo
2023-01-11T21:23:01.6344544Z  distributed/test_c10d_common
2023-01-11T21:23:01.6344807Z  distributed/_tensor/test_dtensor
2023-01-11T21:23:01.6345084Z  distributed/test_pg_wrapper
2023-01-11T21:23:01.6345361Z  distributed/fsdp/test_wrap
2023-01-11T21:23:01.6345665Z  distributed/_shard/sharded_tensor/test_sharded_tensor
2023-01-11T21:23:01.6345993Z  distributed/fsdp/test_fsdp_state_dict
2023-01-11T21:23:01.6346280Z  distributed/test_c10d_nccl
2023-01-11T21:23:03.7802863Z Ignoring disabled issues:  []
2023-01-11T21:23:03.7850966Z Ignoring disabled issues:  []
2023-01-11T21:23:04.1867308Z Running distributed/checkpoint/test_2d_model_state_checkpoint ... [2023-01-11 21:23:04.186264]
2023-01-11T21:23:04.1871105Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_2d_model_state_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:23:04.186786]
2023-01-11T21:23:10.5231625Z 
2023-01-11T21:23:10.5232944Z Expand the folded group to see the log file of distributed/checkpoint/test_2d_model_state_checkpoint
2023-01-11T21:23:10.5239949Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_2d_model_state_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_2d_model_state_checkpoint_29na5b27)
2023-01-11T21:23:10.5240386Z 
2023-01-11T21:23:10.5240505Z Running tests...
2023-01-11T21:23:10.5241044Z ----------------------------------------------------------------------
2023-01-11T21:23:10.5241677Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_2d_model_state_checkpoint
2023-01-11T21:23:10.5242248Z   test_2d_model_state_checkpoint (__main__.Test2dModelStateCheckpoint) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:23:10.5242749Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 901
2023-01-11T21:23:10.5243193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 902
2023-01-11T21:23:10.5243790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:23:10.5244240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:10.5244817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:10.5245270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:10.5245851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:23:10.5246300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:10.5246865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:10.5247311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:10.5247751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:23:10.5248244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:23:10.5248709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:23:10.5249188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:23:10.5249843Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:10.5250541Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:10.5250963Z skip: Need at least 4 CUDA devices (3.975s)
2023-01-11T21:23:10.5251158Z 
2023-01-11T21:23:10.5251428Z ----------------------------------------------------------------------
2023-01-11T21:23:10.5251753Z Ran 1 test in 3.975s
2023-01-11T21:23:10.5251913Z 
2023-01-11T21:23:10.5252024Z OK (skipped=1)
2023-01-11T21:23:10.5252158Z 
2023-01-11T21:23:10.5252283Z Generating XML reports...
2023-01-11T21:23:10.5252957Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_2d_model_state_checkpoint/TEST-Test2dModelStateCheckpoint-20230111212306.xml
2023-01-11T21:23:10.5253367Z 
2023-01-11T21:23:10.5253691Z ##[endgroup]
2023-01-11T21:23:10.5254597Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_2d_model_state_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_2d_model_state_checkpoint_29na5b27)
2023-01-11T21:23:10.5255130Z 
2023-01-11T21:23:10.5255407Z Running distributed/_tensor/test_device_mesh ... [2023-01-11 21:23:10.523268]
2023-01-11T21:23:10.5256090Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_device_mesh.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:23:10.523569]
2023-01-11T21:24:02.8534844Z 
2023-01-11T21:24:02.8535543Z Expand the folded group to see the log file of distributed/_tensor/test_device_mesh
2023-01-11T21:24:02.8536486Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_device_mesh (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_device_mesh_2dxiafo7)
2023-01-11T21:24:02.8538156Z 
2023-01-11T21:24:02.8538914Z Running tests...
2023-01-11T21:24:02.8539784Z ----------------------------------------------------------------------
2023-01-11T21:24:02.8540879Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_device_mesh
2023-01-11T21:24:02.8541918Z   test_all_gather_1d (__main__.DeviceMeshCollectiveTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:24:02.8542825Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1006
2023-01-11T21:24:02.8543699Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1007
2023-01-11T21:24:02.8544557Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1008
2023-01-11T21:24:02.8545344Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1009
2023-01-11T21:24:02.8546112Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 1010
2023-01-11T21:24:02.8546919Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 1011
2023-01-11T21:24:02.8547737Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 1012
2023-01-11T21:24:02.8548509Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 1013
2023-01-11T21:24:02.8549746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8550615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8551332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8551799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8552979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8553435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8554439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8554920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8555518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8555974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8556539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8556989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8557564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8558007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8558583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8559047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8559620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8560413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8560984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8561464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8562038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8562487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8563032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8563506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8564083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8564519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8565084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8565520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8566287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8566739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8567320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8567778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8568218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T21:24:02.8568679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:02.8569148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T21:24:02.8569610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:02.8570062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:02.8570526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:02.8570990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T21:24:02.8571450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T21:24:02.8571818Z skip: Need at least 8 CUDA devices (4.170s)
2023-01-11T21:24:02.8572303Z   test_all_gather_nd (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1278
2023-01-11T21:24:02.8572850Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1279
2023-01-11T21:24:02.8573294Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1280
2023-01-11T21:24:02.8573701Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1281
2023-01-11T21:24:02.8574129Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 1282
2023-01-11T21:24:02.8574558Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 1283
2023-01-11T21:24:02.8574962Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 1284
2023-01-11T21:24:02.8575397Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 1285
2023-01-11T21:24:02.8576003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8576450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8577132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8577611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8578186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8578607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8579174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8579630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8580200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8580616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8581184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8581645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8582214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8582629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8583193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8583645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8584194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8584630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8585189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8585651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8586199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8586644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8587200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8587621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8588187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8588646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8589228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8589670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8590239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8590677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8591304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8591749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8592678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:02.8593149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:02.8593594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T21:24:02.8594049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:02.8594706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:02.8595176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T21:24:02.8595614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T21:24:02.8596066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T21:24:02.8596452Z skip: Need at least 8 CUDA devices (2.514s)
2023-01-11T21:24:02.8596922Z   test_all_gather_uneven (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1550
2023-01-11T21:24:02.8597456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1551
2023-01-11T21:24:02.8597896Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1552
2023-01-11T21:24:02.8598330Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1553
2023-01-11T21:24:02.8598754Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 1554
2023-01-11T21:24:02.8599182Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 1555
2023-01-11T21:24:02.8599612Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 1556
2023-01-11T21:24:02.8600019Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 1557
2023-01-11T21:24:02.8600634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8601083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8601657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8602102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8602696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8603141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8603711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8604153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8604723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8605163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8605710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8606170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8606743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8607188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8607735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8608195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8608761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8609182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8609744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8610202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8610768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8611291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8611866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8612325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8612892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8613313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8613878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8614334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8614887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8615330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8615898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8616351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8616766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T21:24:02.8617242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:02.8617710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T21:24:02.8618157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T21:24:02.8618620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T21:24:02.8619076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:02.8619542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:02.8619985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:02.8620369Z skip: Need at least 8 CUDA devices (2.514s)
2023-01-11T21:24:02.8620851Z   test_all_reduce_1d (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1822
2023-01-11T21:24:02.8621383Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1823
2023-01-11T21:24:02.8621809Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1824
2023-01-11T21:24:02.8622245Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1825
2023-01-11T21:24:02.8622681Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 1826
2023-01-11T21:24:02.8623102Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 1827
2023-01-11T21:24:02.8623540Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 1828
2023-01-11T21:24:02.8623978Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 1829
2023-01-11T21:24:02.8624580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8625011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8625577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8626043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8626598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8627043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8627663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8628176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8628734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8629175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8629741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8630184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8630753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8631190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8631748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8632805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8633400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8633891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8634456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8634898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8635470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8635909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8636454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8636919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8637494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8637930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8638475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8638937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8639507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8639927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8640494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8640952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8641399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T21:24:02.8641854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T21:24:02.8642314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:02.8642779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T21:24:02.8643245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:02.8643689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T21:24:02.8644148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:02.8644606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:02.8644975Z skip: Need at least 8 CUDA devices (2.614s)
2023-01-11T21:24:02.8645626Z   test_all_reduce_nd (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2094
2023-01-11T21:24:02.8646165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2095
2023-01-11T21:24:02.8646612Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2096
2023-01-11T21:24:02.8647035Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2097
2023-01-11T21:24:02.8647519Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 2098
2023-01-11T21:24:02.8647955Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 2099
2023-01-11T21:24:02.8648374Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 2100
2023-01-11T21:24:02.8648805Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 2101
2023-01-11T21:24:02.8649419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8649872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8650424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8650893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8651469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8651896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8652465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8652927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8653498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8653930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8654498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8654959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8655532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8655958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8656524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8656982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8657534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8657985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8658550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8659008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8659563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8660003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8660566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8661006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8661576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8662016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8662700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8663149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8663722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8664160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8664725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8665168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8665603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:02.8666075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T21:24:02.8666533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T21:24:02.8666988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:02.8667454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T21:24:02.8667916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T21:24:02.8668359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:02.8668816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:02.8669201Z skip: Need at least 8 CUDA devices (2.514s)
2023-01-11T21:24:02.8669661Z   test_all_to_all_1d (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2366
2023-01-11T21:24:02.8670191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2367
2023-01-11T21:24:02.8670649Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2368
2023-01-11T21:24:02.8671087Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2369
2023-01-11T21:24:02.8671507Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 2370
2023-01-11T21:24:02.8672448Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 2371
2023-01-11T21:24:02.8672908Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 2372
2023-01-11T21:24:02.8673324Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 2373
2023-01-11T21:24:02.8673937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8674388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8674961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8675420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8675997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8676443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8676996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8677461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8678035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8678474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8679024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8679645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8680231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8680670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8681222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8681684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8682256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8682679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8683251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8683715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8684285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8684706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8685280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8685738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8686306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8686726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8687290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8687748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8688306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8688750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8689314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8689772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8690188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:02.8690662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T21:24:02.8691177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:02.8691627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T21:24:02.8692095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T21:24:02.8692556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:02.8693021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T21:24:02.8693462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:02.8693844Z skip: Need at least 8 CUDA devices (2.514s)
2023-01-11T21:24:02.8694327Z   test_all_to_all_nd (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2638
2023-01-11T21:24:02.8694841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2639
2023-01-11T21:24:02.8695287Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2640
2023-01-11T21:24:02.8695719Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2641
2023-01-11T21:24:02.8696280Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 2642
2023-01-11T21:24:02.8696705Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 2643
2023-01-11T21:24:02.8697134Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 2644
2023-01-11T21:24:02.8697566Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 2645
2023-01-11T21:24:02.8698155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8698604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8699174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8699638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8700197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8700651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8701221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8701684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8702239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8702682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8703248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8703685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8704257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8704704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8705266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8705712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8706283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8706725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8707276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8707733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8708307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8708747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8709299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8709758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8710330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8710770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8711317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8711773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8712767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8713191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8713920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8714386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8714818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:02.8715276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T21:24:02.8715738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T21:24:02.8716203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:02.8716651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T21:24:02.8717118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T21:24:02.8717575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:02.8718043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:02.8718411Z skip: Need at least 8 CUDA devices (2.615s)
2023-01-11T21:24:02.8718890Z   test_broadcast_1d (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2910
2023-01-11T21:24:02.8719421Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2911
2023-01-11T21:24:02.8719867Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2912
2023-01-11T21:24:02.8720287Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2913
2023-01-11T21:24:02.8720721Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 2914
2023-01-11T21:24:02.8721156Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 2915
2023-01-11T21:24:02.8721572Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 2916
2023-01-11T21:24:02.8722010Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 2917
2023-01-11T21:24:02.8722622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8723072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8723622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8724090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8724664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8725090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8725658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8726128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8726703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8727124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8727692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8728151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8728706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8729144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8729708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8730165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8730824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8731282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8731845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8732304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8732855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8733302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8733871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8734312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8734893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8735329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8735895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8736334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8736905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8737344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8737892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8738350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8738790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:02.8739264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T21:24:02.8739713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T21:24:02.8740177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:02.8740642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T21:24:02.8741105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T21:24:02.8741548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:02.8742001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:02.8742388Z skip: Need at least 8 CUDA devices (2.614s)
2023-01-11T21:24:02.8742854Z   test_broadcast_nd (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3182
2023-01-11T21:24:02.8743388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3183
2023-01-11T21:24:02.8743830Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3184
2023-01-11T21:24:02.8744268Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3185
2023-01-11T21:24:02.8744694Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 3186
2023-01-11T21:24:02.8745128Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 3187
2023-01-11T21:24:02.8745564Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 3188
2023-01-11T21:24:02.8745979Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 3189
2023-01-11T21:24:02.8746582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8747133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8747714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8748163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8748733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8749178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8749727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8750189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8750760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8751201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8751753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8752646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8753232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8753670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8754221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8754684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8755258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8755677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8756256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8756715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8757283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8757934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8758507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8758971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8759545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8759971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8760543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8761005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8761559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8761998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8762560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8763018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8763435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:02.8763914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T21:24:02.8764376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:02.8764988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:02.8765458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T21:24:02.8765912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:02.8766369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T21:24:02.8766809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T21:24:02.8767189Z skip: Need at least 8 CUDA devices (2.514s)
2023-01-11T21:24:02.8767680Z   test_reduce_scatter_1d (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3454
2023-01-11T21:24:02.8768220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3455
2023-01-11T21:24:02.8768650Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3456
2023-01-11T21:24:02.8769078Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3457
2023-01-11T21:24:02.8769514Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 3458
2023-01-11T21:24:02.8769932Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 3459
2023-01-11T21:24:02.8770360Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 3460
2023-01-11T21:24:02.8770820Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 3461
2023-01-11T21:24:02.8771425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8771859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8772429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8772899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8773454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8773893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8774463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8774925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8775477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8775914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8776479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8776928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8777509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8777947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8778514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8778953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8779522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8779959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8780522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8780961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8781652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8782094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8782642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8783100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8783664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8784099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8784648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8785105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8785682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8786108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8786673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8787128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8787558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T21:24:02.8788012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:02.8788473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:02.8788939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T21:24:02.8789398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T21:24:02.8789851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:02.8790308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:02.8790762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T21:24:02.8791193Z skip: Need at least 8 CUDA devices (2.514s)
2023-01-11T21:24:02.8791680Z   test_reduce_scatter_nd (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3726
2023-01-11T21:24:02.8792628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3727
2023-01-11T21:24:02.8793075Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3728
2023-01-11T21:24:02.8793495Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3729
2023-01-11T21:24:02.8793925Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 3730
2023-01-11T21:24:02.8794370Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 3731
2023-01-11T21:24:02.8794790Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 3732
2023-01-11T21:24:02.8795220Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 3733
2023-01-11T21:24:02.8795843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8796290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8796841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8797307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8797881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8798419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8799046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8799514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8800089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8800568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8801132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8801597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8802167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8802592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8803160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8803617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8804167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8804603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8805163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8805617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8806168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8806610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8807179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8807621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8808187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8808633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8809197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8809634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8810205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8810642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8811206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8811650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8812084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T21:24:02.8812551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:02.8813000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:02.8813468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T21:24:02.8813924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:02.8814386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:02.8814830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T21:24:02.8815408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T21:24:02.8815802Z skip: Need at least 8 CUDA devices (2.513s)
2023-01-11T21:24:02.8816282Z   test_reduce_scatter_uneven (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3998
2023-01-11T21:24:02.8816824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3999
2023-01-11T21:24:02.8817266Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4000
2023-01-11T21:24:02.8817703Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4001
2023-01-11T21:24:02.8818120Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 4002
2023-01-11T21:24:02.8818553Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 4003
2023-01-11T21:24:02.8818990Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 4004
2023-01-11T21:24:02.8819416Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 4005
2023-01-11T21:24:02.8820026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8820478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8821048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8821493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8822068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8822512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8823069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8823495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8824069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8824530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8825092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8825549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8826126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8826567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8827113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8827574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8828152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8828577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8829140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8829596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8830163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8830584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8831146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8831606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8832652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8833148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8833719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8834174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8834727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8835162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8835726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8836180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8836597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:02.8837077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T21:24:02.8837593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:02.8838044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:02.8838510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:02.8838969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T21:24:02.8839430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T21:24:02.8839873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T21:24:02.8840254Z skip: Need at least 8 CUDA devices (2.514s)
2023-01-11T21:24:02.8840736Z   test_scatter_1d (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4270
2023-01-11T21:24:02.8841271Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4271
2023-01-11T21:24:02.8841697Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4272
2023-01-11T21:24:02.8842130Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4273
2023-01-11T21:24:02.8842566Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 4274
2023-01-11T21:24:02.8842986Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 4275
2023-01-11T21:24:02.8843416Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 4276
2023-01-11T21:24:02.8843849Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 4277
2023-01-11T21:24:02.8844454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8844882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8845461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8845924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8846480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8846919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8847480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8847940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8848495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8848939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8849611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8850060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8850630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8851070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8851630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8852072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8852642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8853081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8853652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8854097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8854669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8855105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8855648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8856103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8856667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8857102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8857647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8858112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8858675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8859090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8859648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8860104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8860535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T21:24:02.8860990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T21:24:02.8861451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T21:24:02.8861922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:02.8862386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:02.8862832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:02.8863286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:02.8863747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T21:24:02.8864114Z skip: Need at least 8 CUDA devices (2.514s)
2023-01-11T21:24:02.8864588Z   test_scatter_nd (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4542
2023-01-11T21:24:02.8865115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4543
2023-01-11T21:24:02.8865558Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4544
2023-01-11T21:24:02.8866097Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4545
2023-01-11T21:24:02.8866532Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 4546
2023-01-11T21:24:02.8866970Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 4547
2023-01-11T21:24:02.8867388Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 4548
2023-01-11T21:24:02.8867817Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 4549
2023-01-11T21:24:02.8868428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8868873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8869428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8869892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8870473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8870896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8871461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8872437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8873156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8873575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8874135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8874592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8875173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8875593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8876156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8876611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8877161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8877595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8878157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8878611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8879163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8879612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8880175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8880617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8881185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8881627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8882185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8882624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8883195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8883796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8884373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8884815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8885253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T21:24:02.8885720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T21:24:02.8886170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T21:24:02.8886625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:02.8887087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:02.8887549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T21:24:02.8888001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:02.8888455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:02.8888840Z skip: Need at least 8 CUDA devices (2.614s)
2023-01-11T21:24:02.8889308Z   test_scatter_uneven (__main__.DeviceMeshCollectiveTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4814
2023-01-11T21:24:02.8889841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4815
2023-01-11T21:24:02.8890283Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4816
2023-01-11T21:24:02.8890721Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4817
2023-01-11T21:24:02.8891187Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 4818
2023-01-11T21:24:02.8891618Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 4819
2023-01-11T21:24:02.8892061Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 4820
2023-01-11T21:24:02.8892477Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 4821
2023-01-11T21:24:02.8893083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8893530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8894096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8894544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8895117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8895559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8896133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8896577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8897148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8897585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8898131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8898596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8899164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8899602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8900208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8900725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8901295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8901716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8902279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8902777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8903346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8903768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8904321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8904769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8905338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8905787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8906367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8906823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8907377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8907822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8908387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8908851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8909271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:02.8909738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:02.8910206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:02.8910653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T21:24:02.8911119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:02.8911576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T21:24:02.8912394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T21:24:02.8912849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T21:24:02.8913240Z skip: Need at least 8 CUDA devices (2.614s)
2023-01-11T21:24:02.8913696Z   test_device_mesh_2d (__main__.DeviceMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5086
2023-01-11T21:24:02.8914186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5087
2023-01-11T21:24:02.8914627Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5088
2023-01-11T21:24:02.8915059Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5089
2023-01-11T21:24:02.8915498Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 5090
2023-01-11T21:24:02.8915916Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 5091
2023-01-11T21:24:02.8916346Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 5092
2023-01-11T21:24:02.8916780Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 5093
2023-01-11T21:24:02.8917537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8917992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8918562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8919027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8919582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8920026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8920581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8921022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8921575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8922043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8922623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8923064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8923638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8924080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8924644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8925085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8925656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8926105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8926654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8927112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8927680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8928117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8928665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8929122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8929692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8930140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8930687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8931144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8931709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8932128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8932693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8933149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8933578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T21:24:02.8934032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:02.8934601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:02.8935073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T21:24:02.8935518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T21:24:02.8935979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T21:24:02.8936436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:02.8936890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:02.8937260Z skip: Need at least 8 CUDA devices (2.514s)
2023-01-11T21:24:02.8937734Z   test_device_mesh_2d_from_dim_groups (__main__.DeviceMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5358
2023-01-11T21:24:02.8938266Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5359
2023-01-11T21:24:02.8938710Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5360
2023-01-11T21:24:02.8939132Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5361
2023-01-11T21:24:02.8939564Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 5362
2023-01-11T21:24:02.8939998Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 5363
2023-01-11T21:24:02.8940412Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 5364
2023-01-11T21:24:02.8940841Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 5365
2023-01-11T21:24:02.8941450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8941893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8942453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8942912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8943488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8943910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8944475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8944932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8945499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8945921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8946481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8946926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8947476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8947939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8948517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8948976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8949531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8949972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8950535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8951093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8951653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8952456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8953037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8953476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8954048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8954487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8955048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8955498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8956069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8956503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8957078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8957542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8957958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T21:24:02.8958428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:02.8958895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T21:24:02.8959347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:02.8959817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T21:24:02.8960277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:02.8960737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:02.8961183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T21:24:02.8961567Z skip: Need at least 8 CUDA devices (2.614s)
2023-01-11T21:24:02.8962040Z   test_device_mesh_dim_groups_error (__main__.DeviceMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5630
2023-01-11T21:24:02.8962566Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5631
2023-01-11T21:24:02.8962989Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5632
2023-01-11T21:24:02.8963418Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5633
2023-01-11T21:24:02.8963867Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 5634
2023-01-11T21:24:02.8964286Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 5635
2023-01-11T21:24:02.8964719Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 5636
2023-01-11T21:24:02.8965156Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 5637
2023-01-11T21:24:02.8965757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8966191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8966760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8967225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8967865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8968385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8968954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8969416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8969972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8970415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8970983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8971426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8971998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8972445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8973010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8973451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8974024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8974464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8975030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8975472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8976039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8976483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8977030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8977491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8978056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8978488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8979038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8979493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8980062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8980485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8981055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8981512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8981943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:02.8982397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:02.8982853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:02.8983319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T21:24:02.8983779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T21:24:02.8984226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T21:24:02.8984785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:02.8985249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T21:24:02.8985622Z skip: Need at least 8 CUDA devices (2.515s)
2023-01-11T21:24:02.8986082Z   test_device_mesh_nd (__main__.DeviceMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5902
2023-01-11T21:24:02.8986587Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5903
2023-01-11T21:24:02.8987024Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5904
2023-01-11T21:24:02.8987444Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5905
2023-01-11T21:24:02.8987875Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 5906
2023-01-11T21:24:02.8988319Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 5907
2023-01-11T21:24:02.8988752Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 5908
2023-01-11T21:24:02.8989185Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 5909
2023-01-11T21:24:02.8989794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8990240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8990792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8991313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8992143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8992646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8993230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8993701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8994273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8994700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8995265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8995724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8996295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8996713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8997281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8997746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.8998298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.8998744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.8999307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.8999769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.9000321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.9000763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.9001325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.9001866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.9002490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.9002935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.9003508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.9003949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.9004525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:24:02.9004961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:24:02.9005529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:24:02.9005971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:24:02.9006415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:24:02.9006896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T21:24:02.9007342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:24:02.9007803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T21:24:02.9008270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:24:02.9008733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T21:24:02.9009175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:24:02.9009630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T21:24:02.9010015Z skip: Need at least 8 CUDA devices (2.514s)
2023-01-11T21:24:02.9010213Z 
2023-01-11T21:24:02.9010474Z ----------------------------------------------------------------------
2023-01-11T21:24:02.9010806Z Ran 19 tests in 50.026s
2023-01-11T21:24:02.9010965Z 
2023-01-11T21:24:02.9011074Z OK (skipped=19)
2023-01-11T21:24:02.9011227Z 
2023-01-11T21:24:02.9011353Z Generating XML reports...
2023-01-11T21:24:02.9011955Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_device_mesh/TEST-DeviceMeshCollectiveTest-20230111212312.xml
2023-01-11T21:24:02.9012721Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_device_mesh/TEST-DeviceMeshTest-20230111212312.xml
2023-01-11T21:24:02.9013060Z 
2023-01-11T21:24:02.9013537Z ##[endgroup]
2023-01-11T21:24:02.9014126Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_device_mesh (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_device_mesh_2dxiafo7)
2023-01-11T21:24:02.9014480Z 
2023-01-11T21:24:02.9014762Z Running distributed/fsdp/test_fsdp_optim_state ... [2023-01-11 21:24:02.854482]
2023-01-11T21:24:02.9015467Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_optim_state.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:24:02.854833]
2023-01-11T21:27:52.0979888Z 
2023-01-11T21:27:52.0980558Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_optim_state
2023-01-11T21:27:52.0983103Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_optim_state (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_optim_state_6jv_ht4p)
2023-01-11T21:27:52.0983503Z 
2023-01-11T21:27:52.0983623Z Running tests...
2023-01-11T21:27:52.0985620Z ----------------------------------------------------------------------
2023-01-11T21:27:52.0986326Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state
2023-01-11T21:27:52.0987175Z   test_compatible_with_named_optimizer (__main__.TestFSDPOptimState) ... skip: The test currently fails on CI. (0.002s)
2023-01-11T21:27:52.0988225Z   test_flatten_sharded_optim_state_dict_nested (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.0988990Z Tests :meth:`flatten_sharded_optim_state_dict` for an FSDP-root ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:27:52.0990059Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6209
2023-01-11T21:27:52.0990528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6210
2023-01-11T21:27:52.0991433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.0992189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.0993037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.0993513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.0994369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.0994807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.0997308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.0998056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.0998893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1000016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1000829Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1001506Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1002051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1002519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1003396Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T21:27:52.1003990Z   warnings.warn(
2023-01-11T21:27:52.1004960Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T21:27:52.1005518Z   warnings.warn(
2023-01-11T21:27:52.1005885Z dist init r=0, world=2
2023-01-11T21:27:52.1006188Z dist init r=1, world=2
2023-01-11T21:27:52.1006608Z ok (5.786s)
2023-01-11T21:27:52.1007331Z   test_flatten_sharded_optim_state_dict_transformer (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1008680Z Tests :meth:`flatten_sharded_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6292
2023-01-11T21:27:52.1009601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6293
2023-01-11T21:27:52.1010683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1011481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1012538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1013362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1014406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1015568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1016589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1017371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1018152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1019052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1020214Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1021428Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1022380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1023258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1025036Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T21:27:52.1025617Z   warnings.warn(
2023-01-11T21:27:52.1026378Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T21:27:52.1026922Z   warnings.warn(
2023-01-11T21:27:52.1027169Z dist init r=0, world=2
2023-01-11T21:27:52.1027398Z dist init r=1, world=2
2023-01-11T21:27:52.1027631Z ok (4.611s)
2023-01-11T21:27:52.1027941Z   test_full_optim_state_dict_keys (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1028413Z Tests that the parameter keys returned by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6375
2023-01-11T21:27:52.1028923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6376
2023-01-11T21:27:52.1029531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1029962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1030533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1030999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1031569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1032496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1033095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1033566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1033998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1034498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1035160Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1035856Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1036361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1036833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1037183Z dist init r=0, world=2
2023-01-11T21:27:52.1037564Z dist init r=1, world=2
2023-01-11T21:27:52.1037840Z ok (4.010s)
2023-01-11T21:27:52.1038176Z   test_full_optim_state_dict_nested_invalid (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1038693Z Tests that :meth:`full_optim_state_dict` raises an error when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6458
2023-01-11T21:27:52.1039191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6459
2023-01-11T21:27:52.1039805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1040256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1040828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1041280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1041859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1042305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1042855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1043314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1043767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1044264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1044907Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1045595Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1046120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1046588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1046923Z dist init r=0, world=2
2023-01-11T21:27:52.1047171Z dist init r=1, world=2
2023-01-11T21:27:52.1047405Z ok (4.111s)
2023-01-11T21:27:52.1047689Z   test_optim_input_warning (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1048201Z Tests that passing the ``optim_input`` argument into optimizer state ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6541
2023-01-11T21:27:52.1048735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6542
2023-01-11T21:27:52.1049331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1050298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1050887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1051356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1051914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1052357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1052925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1053389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1053826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1054322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1054983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1055775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1056313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1056782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1057135Z dist init r=0, world=2
2023-01-11T21:27:52.1057364Z dist init r=1, world=2
2023-01-11T21:27:52.1057601Z ok (4.111s)
2023-01-11T21:27:52.1058081Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1058730Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6624
2023-01-11T21:27:52.1059266Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6625
2023-01-11T21:27:52.1059877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1060328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1060881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1061344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1061921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1062365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1062917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1063379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1063841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1064318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1064977Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1065663Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1066184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1066633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1067534Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1068106Z   warnings.warn(
2023-01-11T21:27:52.1068898Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1069426Z   warnings.warn(
2023-01-11T21:27:52.1069673Z dist init r=0, world=2
2023-01-11T21:27:52.1069919Z dist init r=1, world=2
2023-01-11T21:27:52.1070138Z ok (4.211s)
2023-01-11T21:27:52.1070613Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1071274Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6707
2023-01-11T21:27:52.1072537Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6708
2023-01-11T21:27:52.1073194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1073643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1074217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1074685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1075241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1075681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1076249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1076700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1077155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1077649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1078307Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1078978Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1079501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1079973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1080881Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1081428Z   warnings.warn(
2023-01-11T21:27:52.1082218Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1082777Z   warnings.warn(
2023-01-11T21:27:52.1083022Z dist init r=0, world=2
2023-01-11T21:27:52.1083251Z dist init r=1, world=2
2023-01-11T21:27:52.1083486Z ok (4.211s)
2023-01-11T21:27:52.1083962Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1084608Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6790
2023-01-11T21:27:52.1085147Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6791
2023-01-11T21:27:52.1085758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1086206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1086762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1087232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1087805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1088227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1088795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1089393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1089854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1090337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1091002Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1091693Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1092218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1092672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1093573Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1094140Z   warnings.warn(
2023-01-11T21:27:52.1094933Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1095461Z   warnings.warn(
2023-01-11T21:27:52.1095706Z dist init r=1, world=2
2023-01-11T21:27:52.1095952Z dist init r=0, world=2
2023-01-11T21:27:52.1096167Z ok (4.211s)
2023-01-11T21:27:52.1096642Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1097297Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6873
2023-01-11T21:27:52.1097829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6874
2023-01-11T21:27:52.1098422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1098867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1099438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1099905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1100464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1100902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1101469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1101926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1102380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1102879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1103538Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1104210Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1104731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1105201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1106159Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1106754Z   warnings.warn(
2023-01-11T21:27:52.1107542Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1108155Z   warnings.warn(
2023-01-11T21:27:52.1108401Z dist init r=0, world=2
2023-01-11T21:27:52.1108629Z dist init r=1, world=2
2023-01-11T21:27:52.1108862Z ok (4.311s)
2023-01-11T21:27:52.1109343Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1109996Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6956
2023-01-11T21:27:52.1110532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6957
2023-01-11T21:27:52.1111144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1111597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1112676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1113153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1113728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1114153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1114729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1115193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1115650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1116130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1116783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1117477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1117998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1118452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1119350Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1119927Z   warnings.warn(
2023-01-11T21:27:52.1120712Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1121240Z   warnings.warn(
2023-01-11T21:27:52.1121487Z dist init r=0, world=2
2023-01-11T21:27:52.1121733Z dist init r=1, world=2
2023-01-11T21:27:52.1121953Z ok (4.213s)
2023-01-11T21:27:52.1122425Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1123264Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7039
2023-01-11T21:27:52.1123804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7040
2023-01-11T21:27:52.1124407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1124856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1125429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1125881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1126462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1126904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1127479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1127925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1128379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1128873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1129534Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1130208Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1130732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1131202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1132117Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1132659Z   warnings.warn(
2023-01-11T21:27:52.1133447Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1134002Z   warnings.warn(
2023-01-11T21:27:52.1134228Z dist init r=1, world=2
2023-01-11T21:27:52.1134481Z dist init r=0, world=2
2023-01-11T21:27:52.1134718Z ok (4.311s)
2023-01-11T21:27:52.1135195Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1135849Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7122
2023-01-11T21:27:52.1136381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7123
2023-01-11T21:27:52.1136987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1137416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1137988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1138450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1139026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1139451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1140137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1140606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1141058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1141542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1142209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1142899Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1143400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1143870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1144772Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1145336Z   warnings.warn(
2023-01-11T21:27:52.1146106Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1146656Z   warnings.warn(
2023-01-11T21:27:52.1146903Z dist init r=1, world=2
2023-01-11T21:27:52.1147149Z dist init r=0, world=2
2023-01-11T21:27:52.1147363Z ok (4.211s)
2023-01-11T21:27:52.1147833Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1148503Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7205
2023-01-11T21:27:52.1149037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7206
2023-01-11T21:27:52.1149633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1150081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1150652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1151101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1151675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1152763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1153368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1153815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1154267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1154765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1155410Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1156102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1156625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1157098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1158166Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1158744Z   warnings.warn(
2023-01-11T21:27:52.1159535Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1160084Z   warnings.warn(
2023-01-11T21:27:52.1160310Z dist init r=1, world=2
2023-01-11T21:27:52.1160557Z dist init r=0, world=2
2023-01-11T21:27:52.1160791Z ok (4.215s)
2023-01-11T21:27:52.1161258Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1161931Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7288
2023-01-11T21:27:52.1162465Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7289
2023-01-11T21:27:52.1163077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1163504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1164079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1164544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1165118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1165545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1166117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1166580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1167032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1167510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1168166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1168858Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1169364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1169843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1170710Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T21:27:52.1171261Z   warnings.warn(
2023-01-11T21:27:52.1172007Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T21:27:52.1172551Z   warnings.warn(
2023-01-11T21:27:52.1172795Z dist init r=0, world=2
2023-01-11T21:27:52.1173042Z dist init r=1, world=2
2023-01-11T21:27:52.1173258Z ok (4.311s)
2023-01-11T21:27:52.1173734Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1174507Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7371
2023-01-11T21:27:52.1175028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7372
2023-01-11T21:27:52.1175642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1176092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1176664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1177110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1177687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1178136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1178710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1179153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1179606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1180103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1180745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1181437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1181958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1182436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1183291Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T21:27:52.1183836Z   warnings.warn(
2023-01-11T21:27:52.1184586Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T21:27:52.1185132Z   warnings.warn(
2023-01-11T21:27:52.1185360Z dist init r=0, world=2
2023-01-11T21:27:52.1185605Z dist init r=1, world=2
2023-01-11T21:27:52.1185842Z ok (4.211s)
2023-01-11T21:27:52.1186306Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1186978Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7454
2023-01-11T21:27:52.1187510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7455
2023-01-11T21:27:52.1188115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1188545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1189116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1189581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1190161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1190585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1191266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1191736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1192602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1193107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1193782Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1194470Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1194977Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1195454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1195813Z dist init r=1, world=2
2023-01-11T21:27:52.1196044Z dist init r=0, world=2
2023-01-11T21:27:52.1196280Z ok (3.510s)
2023-01-11T21:27:52.1196761Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1197427Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7533
2023-01-11T21:27:52.1197939Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7534
2023-01-11T21:27:52.1198549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1199001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1199577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1200034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1200618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1201065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1201618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1202080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1202532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1203026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1203672Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1204367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1204888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1205362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1205700Z dist init r=1, world=2
2023-01-11T21:27:52.1205946Z dist init r=0, world=2
2023-01-11T21:27:52.1206185Z ok (3.610s)
2023-01-11T21:27:52.1206648Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1207316Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7612
2023-01-11T21:27:52.1208066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7613
2023-01-11T21:27:52.1208695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1209125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1209699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1210164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1210723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1211168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1211732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1212194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1212637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1213133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1213792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1214469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1214998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1215466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1216339Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T21:27:52.1216875Z   warnings.warn(
2023-01-11T21:27:52.1217628Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T21:27:52.1218176Z   warnings.warn(
2023-01-11T21:27:52.1218422Z dist init r=0, world=2
2023-01-11T21:27:52.1218652Z dist init r=1, world=2
2023-01-11T21:27:52.1218888Z ok (4.311s)
2023-01-11T21:27:52.1219367Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1220019Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7695
2023-01-11T21:27:52.1220557Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7696
2023-01-11T21:27:52.1221166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1221616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1222172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1222636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1223216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1223660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1224208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1224667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1225248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1225733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1226398Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1227090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1227613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1228067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1228934Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T21:27:52.1229488Z   warnings.warn(
2023-01-11T21:27:52.1230242Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T21:27:52.1230768Z   warnings.warn(
2023-01-11T21:27:52.1231015Z dist init r=1, world=2
2023-01-11T21:27:52.1231261Z dist init r=0, world=2
2023-01-11T21:27:52.1231481Z ok (4.311s)
2023-01-11T21:27:52.1232388Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1233079Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7778
2023-01-11T21:27:52.1233618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7779
2023-01-11T21:27:52.1234223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1234676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1235248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1235715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1236273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1236713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1237280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1237724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1238184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1238680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1239339Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1240013Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1240538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1241012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1241366Z dist init r=1, world=2
2023-01-11T21:27:52.1241598Z dist init r=0, world=2
2023-01-11T21:27:52.1241834Z ok (3.509s)
2023-01-11T21:27:52.1242415Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1243135Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7857
2023-01-11T21:27:52.1243664Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7858
2023-01-11T21:27:52.1244277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1244731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1245286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1245749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1246326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1246759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1247329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1247792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1248245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1248889Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1249426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1250076Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1250596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1251057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1251408Z dist init r=1, world=2
2023-01-11T21:27:52.1251659Z dist init r=0, world=2
2023-01-11T21:27:52.1251876Z ok (3.509s)
2023-01-11T21:27:52.1252307Z   test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1252909Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7936
2023-01-11T21:27:52.1253428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7937
2023-01-11T21:27:52.1254023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1254473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1255053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1255505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1256079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1256521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1257092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1257540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1257992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1258491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1259245Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1259942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1260466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1260937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1261824Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1262387Z   warnings.warn(
2023-01-11T21:27:52.1263175Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1263734Z   warnings.warn(
2023-01-11T21:27:52.1263962Z dist init r=0, world=2
2023-01-11T21:27:52.1264209Z dist init r=1, world=2
2023-01-11T21:27:52.1264444Z ok (4.211s)
2023-01-11T21:27:52.1264854Z   test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1265459Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8019
2023-01-11T21:27:52.1265978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8020
2023-01-11T21:27:52.1266589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1267020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1267604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1268072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1268646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1269072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1269638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1270097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1270532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1271029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1271688Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1272870Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1273382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1273853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1274757Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1275313Z   warnings.warn(
2023-01-11T21:27:52.1276188Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1276823Z   warnings.warn(
2023-01-11T21:27:52.1277071Z dist init r=0, world=2
2023-01-11T21:27:52.1277318Z dist init r=1, world=2
2023-01-11T21:27:52.1277537Z ok (4.311s)
2023-01-11T21:27:52.1277969Z   test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1278581Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8102
2023-01-11T21:27:52.1279084Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8103
2023-01-11T21:27:52.1279703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1280151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1280730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1281183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1281758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1282200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1282750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1283213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1283663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1284157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1284803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1285496Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1286018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1286489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1287341Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T21:27:52.1287887Z   warnings.warn(
2023-01-11T21:27:52.1288635Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T21:27:52.1289184Z   warnings.warn(
2023-01-11T21:27:52.1289413Z dist init r=0, world=2
2023-01-11T21:27:52.1289661Z dist init r=1, world=2
2023-01-11T21:27:52.1289896Z ok (4.111s)
2023-01-11T21:27:52.1290310Z   test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1290920Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8185
2023-01-11T21:27:52.1291435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8186
2023-01-11T21:27:52.1292047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1292477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1293106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1293621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1294183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1294625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1295191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1295648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1296081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1296579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1297301Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1298003Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1298512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1298981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1299851Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T21:27:52.1300404Z   warnings.warn(
2023-01-11T21:27:52.1301146Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T21:27:52.1301694Z   warnings.warn(
2023-01-11T21:27:52.1301944Z dist init r=1, world=2
2023-01-11T21:27:52.1302175Z dist init r=0, world=2
2023-01-11T21:27:52.1302415Z ok (4.111s)
2023-01-11T21:27:52.1302730Z   test_rekey_optim_state_dict_to_names (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1303220Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8268
2023-01-11T21:27:52.1303738Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8269
2023-01-11T21:27:52.1304342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1304789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1305341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1305804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1306386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1306829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1307381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1307880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1308336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1308817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1309479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1310168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1310800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1311264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1312572Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1313161Z   warnings.warn(
2023-01-11T21:27:52.1313963Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1314492Z   warnings.warn(
2023-01-11T21:27:52.1314738Z dist init r=1, world=2
2023-01-11T21:27:52.1314990Z dist init r=0, world=2
2023-01-11T21:27:52.1315210Z ok (4.311s)
2023-01-11T21:27:52.1315599Z   test_save_load_without_0th_param_state_state_dict_type_StateDictType_FULL_STATE_DICT (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1316185Z Tests saving and loading an optim state dict for Adam optimizer (i.e. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8351
2023-01-11T21:27:52.1316721Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8352
2023-01-11T21:27:52.1317316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1317763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1318333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1318783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1319372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1319814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1320380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1320823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1321276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1321770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1322427Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1323103Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1323632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1324103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1324441Z dist init r=0, world=2
2023-01-11T21:27:52.1324691Z dist init r=1, world=2
2023-01-11T21:27:52.1324930Z ok (4.011s)
2023-01-11T21:27:52.1325310Z   test_save_load_without_0th_param_state_state_dict_type_StateDictType_SHARDED_STATE_DICT (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1325897Z Tests saving and loading an optim state dict for Adam optimizer (i.e. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8434
2023-01-11T21:27:52.1326427Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8435
2023-01-11T21:27:52.1327032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1327570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1328213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1328692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1329270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1329697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1330261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1330720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1331153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1331647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1332311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1333004Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1333510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1333983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1334851Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T21:27:52.1335405Z   warnings.warn(
2023-01-11T21:27:52.1336146Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T21:27:52.1336689Z   warnings.warn(
2023-01-11T21:27:52.1336936Z dist init r=0, world=2
2023-01-11T21:27:52.1337182Z dist init r=1, world=2
2023-01-11T21:27:52.1337399Z ok (4.011s)
2023-01-11T21:27:52.1337748Z   test_scatter_full_optim_state_dict_nested_halve_world_size (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1338427Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8517
2023-01-11T21:27:52.1338940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8518
2023-01-11T21:27:52.1339540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1339982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1340558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1341010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1341582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1342021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1342569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1343030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1343479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1343972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1344680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1345426Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1345948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1346423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1346893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:27:52.1347385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:27:52.1348043Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:27:52.1348855Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:288: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2023-01-11T21:27:52.1349347Z   warnings.warn(
2023-01-11T21:27:52.1349878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:27:52.1350825Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1351367Z   warnings.warn(
2023-01-11T21:27:52.1352534Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1353117Z   warnings.warn(
2023-01-11T21:27:52.1353498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:27:52.1353984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:27:52.1354653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:27:52.1355337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:27:52.1355738Z dist init r=0, world=2
2023-01-11T21:27:52.1355968Z dist init r=1, world=2
2023-01-11T21:27:52.1356204Z ok (4.411s)
2023-01-11T21:27:52.1356638Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1357376Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8610
2023-01-11T21:27:52.1357917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8611
2023-01-11T21:27:52.1358520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1358969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1359521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1359988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1360565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1361008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1361557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1362016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1362646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1363137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1363800Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1364492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1365013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1365471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1366370Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1366940Z   warnings.warn(
2023-01-11T21:27:52.1367726Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1368249Z   warnings.warn(
2023-01-11T21:27:52.1368497Z dist init r=0, world=2
2023-01-11T21:27:52.1368745Z dist init r=1, world=2
2023-01-11T21:27:52.1368962Z ok (4.411s)
2023-01-11T21:27:52.1369395Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1370142Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8693
2023-01-11T21:27:52.1370680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8694
2023-01-11T21:27:52.1371267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1371713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1372281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1372749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1373303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1373743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1374306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1374749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1375212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1375707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1376361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1377038Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1377558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1378027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1378987Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1379589Z   warnings.warn(
2023-01-11T21:27:52.1380377Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1380933Z   warnings.warn(
2023-01-11T21:27:52.1381179Z dist init r=1, world=2
2023-01-11T21:27:52.1381409Z dist init r=0, world=2
2023-01-11T21:27:52.1381644Z ok (4.513s)
2023-01-11T21:27:52.1382074Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1382807Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8776
2023-01-11T21:27:52.1383341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8777
2023-01-11T21:27:52.1383953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1384403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1384960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1385424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1386002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1386425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1386990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1387451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1387908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1388387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1389042Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1389734Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1390261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1390720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1391616Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1392590Z   warnings.warn(
2023-01-11T21:27:52.1393401Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1393930Z   warnings.warn(
2023-01-11T21:27:52.1394176Z dist init r=1, world=2
2023-01-11T21:27:52.1394428Z dist init r=0, world=2
2023-01-11T21:27:52.1394647Z ok (4.411s)
2023-01-11T21:27:52.1395075Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1395825Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8859
2023-01-11T21:27:52.1396360Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8860
2023-01-11T21:27:52.1397154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1397623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1398202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1398654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1399231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1399672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1400245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1400694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1401155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1401651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1402294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1402987Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1403512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1403986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1404874Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1405446Z   warnings.warn(
2023-01-11T21:27:52.1406240Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1406793Z   warnings.warn(
2023-01-11T21:27:52.1407021Z dist init r=1, world=2
2023-01-11T21:27:52.1407270Z dist init r=0, world=2
2023-01-11T21:27:52.1407505Z ok (4.411s)
2023-01-11T21:27:52.1407965Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1408721Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8942
2023-01-11T21:27:52.1409253Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8943
2023-01-11T21:27:52.1409862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1410293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1410867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1411336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1411910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1412337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1412903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1413362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1413908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1414419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1415082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1415777Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1416280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1416753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1417659Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1418231Z   warnings.warn(
2023-01-11T21:27:52.1419008Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1419559Z   warnings.warn(
2023-01-11T21:27:52.1419805Z dist init r=0, world=2
2023-01-11T21:27:52.1420051Z dist init r=1, world=2
2023-01-11T21:27:52.1420269Z ok (4.511s)
2023-01-11T21:27:52.1420694Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1421441Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9025
2023-01-11T21:27:52.1421956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9026
2023-01-11T21:27:52.1422569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1423015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1423587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1424041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1424620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1425063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1425609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1426072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1426532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1427030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1427679Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1428368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1428889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1429356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1430243Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1430911Z   warnings.warn(
2023-01-11T21:27:52.1431712Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1432697Z   warnings.warn(
2023-01-11T21:27:52.1432932Z dist init r=0, world=2
2023-01-11T21:27:52.1433181Z dist init r=1, world=2
2023-01-11T21:27:52.1433420Z ok (4.511s)
2023-01-11T21:27:52.1433834Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1434589Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9108
2023-01-11T21:27:52.1435129Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9109
2023-01-11T21:27:52.1435745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1436179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1436749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1437215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1437772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1438217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1438784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1439242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1439688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1440189Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1440845Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1441539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1442043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1442515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1443415Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1443990Z   warnings.warn(
2023-01-11T21:27:52.1444754Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1445317Z   warnings.warn(
2023-01-11T21:27:52.1445563Z dist init r=0, world=2
2023-01-11T21:27:52.1445791Z dist init r=1, world=2
2023-01-11T21:27:52.1446025Z ok (4.411s)
2023-01-11T21:27:52.1446448Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1447197Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9191
2023-01-11T21:27:52.1447709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9192
2023-01-11T21:27:52.1448481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1448943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1449506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1449972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1450548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1450991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1451542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1452005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1452462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1452959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1453602Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1454289Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1454814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1455269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1456169Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1456738Z   warnings.warn(
2023-01-11T21:27:52.1457529Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1458074Z   warnings.warn(
2023-01-11T21:27:52.1458299Z dist init r=0, world=2
2023-01-11T21:27:52.1458543Z dist init r=1, world=2
2023-01-11T21:27:52.1458781Z ok (4.411s)
2023-01-11T21:27:52.1459098Z   test_scatter_full_optim_state_dict_transformer (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1459757Z Tests :meth:`scatter_full_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9274
2023-01-11T21:27:52.1460281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9275
2023-01-11T21:27:52.1460870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1461320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1461894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1462361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1462923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1463363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1463928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1464375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1464828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1465434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1466102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1466779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1467303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1467770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1468255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:27:52.1468727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:27:52.1469384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:27:52.1470078Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:27:52.1470901Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:288: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2023-01-11T21:27:52.1471373Z   warnings.warn(
2023-01-11T21:27:52.1472454Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T21:27:52.1473721Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T21:27:52.1474472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:27:52.1474957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:27:52.1475618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:27:52.1476311Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:27:52.1476713Z dist init r=0, world=2
2023-01-11T21:27:52.1476943Z dist init r=1, world=2
2023-01-11T21:27:52.1477182Z ok (4.711s)
2023-01-11T21:27:52.1477530Z   test_shard_full_optim_state_dict_nested_halve_world_size (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1478203Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9367
2023-01-11T21:27:52.1478734Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9368
2023-01-11T21:27:52.1479336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1479782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1480343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1480813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1481386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1481828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1482544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1483016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1483472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1483951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1484614Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1485301Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1485823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1486279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1486763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:27:52.1487254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:27:52.1487908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:27:52.1488722Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:288: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2023-01-11T21:27:52.1489211Z   warnings.warn(
2023-01-11T21:27:52.1489739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:27:52.1490697Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1491248Z   warnings.warn(
2023-01-11T21:27:52.1492037Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1492593Z   warnings.warn(
2023-01-11T21:27:52.1492969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:27:52.1493445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:27:52.1494101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:27:52.1494786Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:27:52.1495183Z dist init r=0, world=2
2023-01-11T21:27:52.1495431Z dist init r=1, world=2
2023-01-11T21:27:52.1495670Z ok (4.411s)
2023-01-11T21:27:52.1496081Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1496845Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9460
2023-01-11T21:27:52.1497384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9461
2023-01-11T21:27:52.1497984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1498416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1498988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1499576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1500164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1500590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1501158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1501618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1502056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1502553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1503209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1503906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1504411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1504880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1505775Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1506337Z   warnings.warn(
2023-01-11T21:27:52.1507116Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1507670Z   warnings.warn(
2023-01-11T21:27:52.1507970Z dist init r=1, world=2
2023-01-11T21:27:52.1508222Z dist init r=0, world=2
2023-01-11T21:27:52.1508440Z ok (4.511s)
2023-01-11T21:27:52.1508868Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1509624Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9543
2023-01-11T21:27:52.1510143Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9544
2023-01-11T21:27:52.1510752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1511198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1511773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1512419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1513011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1513455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1514006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1514472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1514925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1515425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1516072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1516931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1517472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1517944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1518834Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1519404Z   warnings.warn(
2023-01-11T21:27:52.1520198Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1520753Z   warnings.warn(
2023-01-11T21:27:52.1520980Z dist init r=0, world=2
2023-01-11T21:27:52.1521228Z dist init r=1, world=2
2023-01-11T21:27:52.1521465Z ok (4.411s)
2023-01-11T21:27:52.1521873Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1522625Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9626
2023-01-11T21:27:52.1523161Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9627
2023-01-11T21:27:52.1523765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1524196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1524769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1525243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1525805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1526249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1526822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1527285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1527721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1528219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1528878Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1529578Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1530089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1530563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1531469Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1532027Z   warnings.warn(
2023-01-11T21:27:52.1532804Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1533369Z   warnings.warn(
2023-01-11T21:27:52.1533722Z dist init r=0, world=2
2023-01-11T21:27:52.1533993Z dist init r=1, world=2
2023-01-11T21:27:52.1534236Z ok (4.311s)
2023-01-11T21:27:52.1534659Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1535412Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9709
2023-01-11T21:27:52.1535932Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9710
2023-01-11T21:27:52.1536533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1536979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1537534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1538012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1538585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1539023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1539571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1540035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1540486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1540981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1541623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1542026Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1542261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1542488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1543141Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1543256Z   warnings.warn(
2023-01-11T21:27:52.1543905Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1544014Z   warnings.warn(
2023-01-11T21:27:52.1544112Z dist init r=0, world=2
2023-01-11T21:27:52.1544224Z dist init r=1, world=2
2023-01-11T21:27:52.1544325Z ok (4.412s)
2023-01-11T21:27:52.1544627Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1545072Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9792
2023-01-11T21:27:52.1545289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9793
2023-01-11T21:27:52.1545655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1545830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1546189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1546495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1546873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1547046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1547419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1547609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1547858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1548097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1548498Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1548883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1549118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1549345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1549999Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1550112Z   warnings.warn(
2023-01-11T21:27:52.1550759Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1550869Z   warnings.warn(
2023-01-11T21:27:52.1550984Z dist init r=0, world=2
2023-01-11T21:27:52.1551098Z dist init r=1, world=2
2023-01-11T21:27:52.1551180Z ok (4.411s)
2023-01-11T21:27:52.1551479Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1552140Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9875
2023-01-11T21:27:52.1552371Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9876
2023-01-11T21:27:52.1552749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1552924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1553300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1553497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1553866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1554022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1554399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1554593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1554842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1555085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1555480Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1555957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1556245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1556456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1557120Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1557233Z   warnings.warn(
2023-01-11T21:27:52.1557873Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1557982Z   warnings.warn(
2023-01-11T21:27:52.1558096Z dist init r=0, world=2
2023-01-11T21:27:52.1558207Z dist init r=1, world=2
2023-01-11T21:27:52.1558306Z ok (4.511s)
2023-01-11T21:27:52.1558606Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1559036Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9958
2023-01-11T21:27:52.1559257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9959
2023-01-11T21:27:52.1559627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1559802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1560177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1560367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1560741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1560914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1561291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1561463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1561709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1561955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1562354Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1562752Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1562986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1563215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1563872Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1563985Z   warnings.warn(
2023-01-11T21:27:52.1564617Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1564727Z   warnings.warn(
2023-01-11T21:27:52.1564898Z dist init r=1, world=2
2023-01-11T21:27:52.1565045Z dist init r=0, world=2
2023-01-11T21:27:52.1565149Z ok (4.411s)
2023-01-11T21:27:52.1565449Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1565895Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10041
2023-01-11T21:27:52.1566119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10042
2023-01-11T21:27:52.1566488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1566646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1567026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1567215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1567589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1567762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1568138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1568329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1568578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1568799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1569200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1569603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1569836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1570062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1570716Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1570828Z   warnings.warn(
2023-01-11T21:27:52.1571476Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1571585Z   warnings.warn(
2023-01-11T21:27:52.1571699Z dist init r=0, world=2
2023-01-11T21:27:52.1571793Z dist init r=1, world=2
2023-01-11T21:27:52.1571891Z ok (4.311s)
2023-01-11T21:27:52.1572102Z   test_shard_full_optim_state_dict_transformer (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1572527Z Tests :meth:`shard_full_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10124
2023-01-11T21:27:52.1572747Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10125
2023-01-11T21:27:52.1573112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1573285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1573642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1573834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1574320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1574498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1574877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1575066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1575313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1575552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1575950Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1576328Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1576568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1576795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1577038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:27:52.1577273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:27:52.1577670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:27:52.1578062Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:27:52.1578592Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:288: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2023-01-11T21:27:52.1578708Z   warnings.warn(
2023-01-11T21:27:52.1579467Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T21:27:52.1580216Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T21:27:52.1580446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:27:52.1580681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:27:52.1581082Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:27:52.1581481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:27:52.1581591Z dist init r=0, world=2
2023-01-11T21:27:52.1581698Z dist init r=1, world=2
2023-01-11T21:27:52.1581797Z ok (4.812s)
2023-01-11T21:27:52.1582119Z   test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_FULL_STATE_DICT_add_to_fsdp_module_False (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1582414Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10217
2023-01-11T21:27:52.1582634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10218
2023-01-11T21:27:52.1583005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1583281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1583667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1583858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1584222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1584396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1584768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1584937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1585182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1585423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1585829Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1586228Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1586459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1586686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1587339Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1587454Z   warnings.warn(
2023-01-11T21:27:52.1588112Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1588210Z   warnings.warn(
2023-01-11T21:27:52.1588319Z dist init r=1, world=2
2023-01-11T21:27:52.1588427Z dist init r=0, world=2
2023-01-11T21:27:52.1588527Z ok (4.110s)
2023-01-11T21:27:52.1588844Z   test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_FULL_STATE_DICT_add_to_fsdp_module_True (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1589154Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10300
2023-01-11T21:27:52.1589373Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10301
2023-01-11T21:27:52.1589745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1589910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1590289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1590480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1590846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1591020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1591397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1591586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1592006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1592331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1592808Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1593205Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1593437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1593667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1594323Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1594434Z   warnings.warn(
2023-01-11T21:27:52.1595089Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:1093: UserWarning: The `optim_input` argument is deprecated and will be removed after PyTorch 1.13. You may remove it from your code without changing its functionality.
2023-01-11T21:27:52.1595202Z   warnings.warn(
2023-01-11T21:27:52.1595313Z dist init r=1, world=2
2023-01-11T21:27:52.1595404Z dist init r=0, world=2
2023-01-11T21:27:52.1595502Z ok (4.111s)
2023-01-11T21:27:52.1595822Z   test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_SHARDED_STATE_DICT_add_to_fsdp_module_False (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1596133Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10383
2023-01-11T21:27:52.1596352Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10384
2023-01-11T21:27:52.1596788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1596974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1597361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1597535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1597900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1598073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1598448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1598637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1598882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1599124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1599529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1599925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1600137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1600368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1600993Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T21:27:52.1601104Z   warnings.warn(
2023-01-11T21:27:52.1601769Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T21:27:52.1601926Z   warnings.warn(
2023-01-11T21:27:52.1602038Z dist init r=0, world=2
2023-01-11T21:27:52.1602146Z dist init r=1, world=2
2023-01-11T21:27:52.1602227Z ok (4.111s)
2023-01-11T21:27:52.1602551Z   test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_SHARDED_STATE_DICT_add_to_fsdp_module_True (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1602864Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10466
2023-01-11T21:27:52.1603083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10467
2023-01-11T21:27:52.1603454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1603632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1604012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1604203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1604567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1604721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1605093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1605282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1605527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1605766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1606174Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1606573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1606802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1607028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1607635Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T21:27:52.1607749Z   warnings.warn(
2023-01-11T21:27:52.1608419Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T21:27:52.1608534Z   warnings.warn(
2023-01-11T21:27:52.1608645Z dist init r=0, world=2
2023-01-11T21:27:52.1608755Z dist init r=1, world=2
2023-01-11T21:27:52.1608855Z ok (4.111s)
2023-01-11T21:27:52.1609026Z   test_use_orig_params (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1609362Z Tests :meth:`optim_state_dict` for an FSDP-root nested model. ... skip: The test currently fails on CI. (0.001s)
2023-01-11T21:27:52.1609541Z   test_use_orig_params_error (__main__.TestFSDPOptimState)
2023-01-11T21:27:52.1609856Z Tests that the optimizer state checkpointing APIs raise an error ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10549
2023-01-11T21:27:52.1610079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10550
2023-01-11T21:27:52.1610446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1610724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1611110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1611301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1611666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:27:52.1611822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:27:52.1612199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:27:52.1612390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:27:52.1612638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:27:52.1612883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:27:52.1613283Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1613680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:27:52.1613910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:27:52.1614139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:27:52.1614233Z dist init r=1, world=2
2023-01-11T21:27:52.1614341Z dist init r=0, world=2
2023-01-11T21:27:52.1614439Z ok (4.111s)
2023-01-11T21:27:52.1614461Z 
2023-01-11T21:27:52.1614733Z ----------------------------------------------------------------------
2023-01-11T21:27:52.1614849Z Ran 55 tests in 226.668s
2023-01-11T21:27:52.1614872Z 
2023-01-11T21:27:52.1614981Z OK (skipped=2)
2023-01-11T21:27:52.1615003Z 
2023-01-11T21:27:52.1615126Z Generating XML reports...
2023-01-11T21:27:52.1615587Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state/TEST-TestFSDPOptimState-20230111212405.xml
2023-01-11T21:27:52.1615607Z 
2023-01-11T21:27:52.1616050Z ##[endgroup]
2023-01-11T21:27:52.1616544Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_optim_state (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_optim_state_6jv_ht4p)
2023-01-11T21:27:52.1616565Z 
2023-01-11T21:27:52.1616877Z Running distributed/algorithms/quantization/test_quantization ... [2023-01-11 21:27:52.099168]
2023-01-11T21:27:52.1616985Z /usr/bin/mpiexec
2023-01-11T21:27:52.1617211Z MPI not available -- MPI backend tests will be skipped
2023-01-11T21:27:52.1617598Z Map different backends to different shards for distributed/algorithms/quantization/test_quantization: {'gloo': 1, 'nccl': 2}
2023-01-11T21:27:52.1617731Z Shard 2: test should be run in 1
2023-01-11T21:27:52.1617926Z Running distributed tests for the nccl backend with env init_method in shard 2 of 3
2023-01-11T21:27:52.1618483Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:27:52.101122]
2023-01-11T21:28:28.6830094Z 
2023-01-11T21:28:28.6830865Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2023-01-11T21:28:28.6832456Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_ddpag5wm)
2023-01-11T21:28:28.6833875Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2023-01-11T21:28:28.6834859Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2023-01-11T21:28:28.6837888Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2023-01-11T21:28:28.6838298Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2023-01-11T21:28:28.6838641Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2023-01-11T21:28:28.6839011Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2023-01-11T21:28:28.6839391Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2023-01-11T21:28:28.6839744Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:28:28.6840393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:28:28.6840848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:28.6841445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:28.6841904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:28.6842135Z 
2023-01-11T21:28:28.6842245Z Running tests...
2023-01-11T21:28:28.6842654Z ----------------------------------------------------------------------
2023-01-11T21:28:28.6843265Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2023-01-11T21:28:28.6843814Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s)
2023-01-11T21:28:28.6844326Z 
2023-01-11T21:28:28.6844609Z ----------------------------------------------------------------------
2023-01-11T21:28:28.6844940Z Ran 1 test in 0.001s
2023-01-11T21:28:28.6845099Z 
2023-01-11T21:28:28.6845212Z OK (skipped=1)
2023-01-11T21:28:28.6845348Z 
2023-01-11T21:28:28.6845477Z Generating XML reports...
2023-01-11T21:28:28.6846140Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212755.xml
2023-01-11T21:28:28.6846889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:28:28.6847324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:28.6847893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:28.6848359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:28.6848768Z 
2023-01-11T21:28:28.6848894Z Running tests...
2023-01-11T21:28:28.6849479Z ----------------------------------------------------------------------
2023-01-11T21:28:28.6850088Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2023-01-11T21:28:28.6850669Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s)
2023-01-11T21:28:28.6850957Z 
2023-01-11T21:28:28.6851227Z ----------------------------------------------------------------------
2023-01-11T21:28:28.6851533Z Ran 1 test in 0.001s
2023-01-11T21:28:28.6851695Z 
2023-01-11T21:28:28.6851805Z OK (skipped=1)
2023-01-11T21:28:28.6851954Z 
2023-01-11T21:28:28.6852150Z Generating XML reports...
2023-01-11T21:28:28.6852800Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212757.xml
2023-01-11T21:28:28.6853546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:28:28.6853971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:28.6854541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:28.6855197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:28.6855428Z 
2023-01-11T21:28:28.6855538Z Running tests...
2023-01-11T21:28:28.6855925Z ----------------------------------------------------------------------
2023-01-11T21:28:28.6856525Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2023-01-11T21:28:28.6857065Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:28:28.6857516Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10767
2023-01-11T21:28:28.6857962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10768
2023-01-11T21:28:28.6858562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:28:28.6859005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:28.6859565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:28.6860024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:28.6860591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:28:28.6861027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:28.6861571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:28.6862028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:28.6862459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:28.6862920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:28.6863404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:28.6863908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:28.6864566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:28.6865088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:28:28.6865737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:28.6866262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:28:28.6866911Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:28:28.6867582Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:28:28.6867979Z ok (5.368s)
2023-01-11T21:28:28.6868125Z 
2023-01-11T21:28:28.6868388Z ----------------------------------------------------------------------
2023-01-11T21:28:28.6868690Z Ran 1 test in 5.368s
2023-01-11T21:28:28.6868847Z 
2023-01-11T21:28:28.6868939Z OK
2023-01-11T21:28:28.6869071Z 
2023-01-11T21:28:28.6869193Z Generating XML reports...
2023-01-11T21:28:28.6869856Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212759.xml
2023-01-11T21:28:28.6870598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:28:28.6871043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:28.6871614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:28.6872637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:28.6873052Z 
2023-01-11T21:28:28.6873171Z Running tests...
2023-01-11T21:28:28.6873579Z ----------------------------------------------------------------------
2023-01-11T21:28:28.6874182Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2023-01-11T21:28:28.6874714Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:28:28.6875182Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10886
2023-01-11T21:28:28.6875629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10887
2023-01-11T21:28:28.6876216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:28:28.6876662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:28.6877229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:28.6877703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:28.6878260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:28:28.6878699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:28.6879266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:28.6879723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:28.6880141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:28.6880612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:28.6881097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:28.6881583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:28.6882240Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:28.6882771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:28:28.6883417Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:28.6883930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:28:28.6884568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:28:28.6885242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:28:28.6885638Z ok (5.288s)
2023-01-11T21:28:28.6885767Z 
2023-01-11T21:28:28.6886036Z ----------------------------------------------------------------------
2023-01-11T21:28:28.6886356Z Ran 1 test in 5.288s
2023-01-11T21:28:28.6886513Z 
2023-01-11T21:28:28.6886605Z OK
2023-01-11T21:28:28.6886736Z 
2023-01-11T21:28:28.6886841Z Generating XML reports...
2023-01-11T21:28:28.6887501Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212807.xml
2023-01-11T21:28:28.6888254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:28:28.6888700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:28.6889254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:28.6889715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:28.6890033Z 
2023-01-11T21:28:28.6890139Z Running tests...
2023-01-11T21:28:28.6890567Z ----------------------------------------------------------------------
2023-01-11T21:28:28.6891170Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2023-01-11T21:28:28.6891730Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:28:28.6892207Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11005
2023-01-11T21:28:28.6892636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11006
2023-01-11T21:28:28.6893233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:28:28.6893676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:28.6894227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:28.6894703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:28.6895280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:28:28.6895718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:28.6896267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:28.6896724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:28.6897154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:28.6897644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:28.6898111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:28.6898596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:28.6899261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:28.6899777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:28:28.6900423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:28.6900946Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:28:28.6901589Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:28:28.6902256Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:28:28.6902646Z ok (5.442s)
2023-01-11T21:28:28.6902796Z 
2023-01-11T21:28:28.6903059Z ----------------------------------------------------------------------
2023-01-11T21:28:28.6903383Z Ran 1 test in 5.442s
2023-01-11T21:28:28.6903523Z 
2023-01-11T21:28:28.6903614Z OK
2023-01-11T21:28:28.6903749Z 
2023-01-11T21:28:28.6903869Z Generating XML reports...
2023-01-11T21:28:28.6904529Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212814.xml
2023-01-11T21:28:28.6905271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:28:28.6905715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:28.6906283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:28.6906746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:28.6906959Z 
2023-01-11T21:28:28.6907127Z Running tests...
2023-01-11T21:28:28.6907578Z ----------------------------------------------------------------------
2023-01-11T21:28:28.6908183Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2023-01-11T21:28:28.6908722Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:28:28.6909203Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11124
2023-01-11T21:28:28.6909647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11125
2023-01-11T21:28:28.6910250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:28:28.6910732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:28.6911303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:28.6911768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:28.6913039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:28:28.6913487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:28.6914051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:28.6914511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:28.6914929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:28.6915396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:28.6915882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:28.6916381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:28.6917034Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:28.6917573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:28:28.6918221Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:28.6918754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:28:28.6919383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:28:28.6920061Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:28:28.6920449Z ok (5.363s)
2023-01-11T21:28:28.6920596Z 
2023-01-11T21:28:28.6920840Z ----------------------------------------------------------------------
2023-01-11T21:28:28.6921171Z Ran 1 test in 5.363s
2023-01-11T21:28:28.6921329Z 
2023-01-11T21:28:28.6921421Z OK
2023-01-11T21:28:28.6921551Z 
2023-01-11T21:28:28.6921673Z Generating XML reports...
2023-01-11T21:28:28.6922316Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212822.xml
2023-01-11T21:28:28.6922727Z 
2023-01-11T21:28:28.6923102Z ##[endgroup]
2023-01-11T21:28:28.6923812Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_ddpag5wm)
2023-01-11T21:28:28.6924240Z 
2023-01-11T21:28:28.6924452Z Running distributed tests for the nccl backend with file init_method in shard 2 of 3
2023-01-11T21:28:28.6925233Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:28:28.684658]
2023-01-11T21:29:05.2263992Z 
2023-01-11T21:29:05.2264727Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2023-01-11T21:29:05.2267879Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_cx85kh7g)
2023-01-11T21:29:05.2269029Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2023-01-11T21:29:05.2269878Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2023-01-11T21:29:05.2270241Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2023-01-11T21:29:05.2270608Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2023-01-11T21:29:05.2270968Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2023-01-11T21:29:05.2271345Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2023-01-11T21:29:05.2271707Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2023-01-11T21:29:05.2272790Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:29:05.2273468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:29:05.2273903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:29:05.2274487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:29:05.2274957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:29:05.2275194Z 
2023-01-11T21:29:05.2275331Z Running tests...
2023-01-11T21:29:05.2275723Z ----------------------------------------------------------------------
2023-01-11T21:29:05.2276460Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2023-01-11T21:29:05.2277159Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s)
2023-01-11T21:29:05.2277448Z 
2023-01-11T21:29:05.2277718Z ----------------------------------------------------------------------
2023-01-11T21:29:05.2278034Z Ran 1 test in 0.001s
2023-01-11T21:29:05.2278195Z 
2023-01-11T21:29:05.2278305Z OK (skipped=1)
2023-01-11T21:29:05.2278457Z 
2023-01-11T21:29:05.2278580Z Generating XML reports...
2023-01-11T21:29:05.2279219Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212832.xml
2023-01-11T21:29:05.2279980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:29:05.2280439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:29:05.2281037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:29:05.2281485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:29:05.2281717Z 
2023-01-11T21:29:05.2281827Z Running tests...
2023-01-11T21:29:05.2282228Z ----------------------------------------------------------------------
2023-01-11T21:29:05.2282823Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2023-01-11T21:29:05.2283372Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... skip: Only gloo backend supports all_gather_fp16 (0.001s)
2023-01-11T21:29:05.2283661Z 
2023-01-11T21:29:05.2283926Z ----------------------------------------------------------------------
2023-01-11T21:29:05.2284379Z Ran 1 test in 0.001s
2023-01-11T21:29:05.2284749Z 
2023-01-11T21:29:05.2284907Z OK (skipped=1)
2023-01-11T21:29:05.2285068Z 
2023-01-11T21:29:05.2285191Z Generating XML reports...
2023-01-11T21:29:05.2285856Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212834.xml
2023-01-11T21:29:05.2286603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:29:05.2287033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:29:05.2287604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:29:05.2288208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:29:05.2288440Z 
2023-01-11T21:29:05.2288617Z Running tests...
2023-01-11T21:29:05.2289020Z ----------------------------------------------------------------------
2023-01-11T21:29:05.2289618Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2023-01-11T21:29:05.2290291Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:29:05.2290767Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11342
2023-01-11T21:29:05.2291210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11343
2023-01-11T21:29:05.2291938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:29:05.2292385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:29:05.2292954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:29:05.2293419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:29:05.2293977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:29:05.2294419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:29:05.2294983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:29:05.2295423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:29:05.2295857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:29:05.2296340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:29:05.2296819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:29:05.2297283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:29:05.2297938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:29:05.2298476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:29:05.2299117Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:29:05.2299626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:29:05.2300262Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:29:05.2300930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:29:05.2301301Z ok (5.293s)
2023-01-11T21:29:05.2301446Z 
2023-01-11T21:29:05.2301712Z ----------------------------------------------------------------------
2023-01-11T21:29:05.2302034Z Ran 1 test in 5.293s
2023-01-11T21:29:05.2302192Z 
2023-01-11T21:29:05.2302362Z OK
2023-01-11T21:29:05.2302479Z 
2023-01-11T21:29:05.2302644Z Generating XML reports...
2023-01-11T21:29:05.2303310Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212836.xml
2023-01-11T21:29:05.2304064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:29:05.2304493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:29:05.2305064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:29:05.2305526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:29:05.2305753Z 
2023-01-11T21:29:05.2305861Z Running tests...
2023-01-11T21:29:05.2306241Z ----------------------------------------------------------------------
2023-01-11T21:29:05.2306838Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2023-01-11T21:29:05.2307544Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:29:05.2307994Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11461
2023-01-11T21:29:05.2308440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11462
2023-01-11T21:29:05.2309038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:29:05.2309484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:29:05.2310036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:29:05.2310494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:29:05.2311061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:29:05.2311506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:29:05.2312727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:29:05.2313243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:29:05.2313678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:29:05.2314149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:29:05.2314629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:29:05.2315104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:29:05.2315765Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:29:05.2316289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:29:05.2316930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:29:05.2317445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:29:05.2318080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:29:05.2318741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:29:05.2319129Z ok (5.288s)
2023-01-11T21:29:05.2319275Z 
2023-01-11T21:29:05.2319539Z ----------------------------------------------------------------------
2023-01-11T21:29:05.2319848Z Ran 1 test in 5.288s
2023-01-11T21:29:05.2320006Z 
2023-01-11T21:29:05.2320098Z OK
2023-01-11T21:29:05.2320228Z 
2023-01-11T21:29:05.2320466Z Generating XML reports...
2023-01-11T21:29:05.2321186Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212844.xml
2023-01-11T21:29:05.2321942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:29:05.2322387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:29:05.2322956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:29:05.2323402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:29:05.2323634Z 
2023-01-11T21:29:05.2323743Z Running tests...
2023-01-11T21:29:05.2324140Z ----------------------------------------------------------------------
2023-01-11T21:29:05.2324730Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2023-01-11T21:29:05.2325273Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:29:05.2325748Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11580
2023-01-11T21:29:05.2326193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11581
2023-01-11T21:29:05.2326788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:29:05.2327215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:29:05.2327782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:29:05.2328239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:29:05.2328791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:29:05.2329227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:29:05.2329793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:29:05.2330248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:29:05.2330661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:29:05.2331127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:29:05.2331606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:29:05.2332075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:29:05.2332724Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:29:05.2333248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:29:05.2333895Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:29:05.2334403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:29:05.2335036Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:29:05.2335708Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:29:05.2336096Z ok (5.233s)
2023-01-11T21:29:05.2336224Z 
2023-01-11T21:29:05.2336489Z ----------------------------------------------------------------------
2023-01-11T21:29:05.2336810Z Ran 1 test in 5.233s
2023-01-11T21:29:05.2336968Z 
2023-01-11T21:29:05.2337059Z OK
2023-01-11T21:29:05.2337193Z 
2023-01-11T21:29:05.2337297Z Generating XML reports...
2023-01-11T21:29:05.2338061Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212851.xml
2023-01-11T21:29:05.2338818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:29:05.2339261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:29:05.2339812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:29:05.2340276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:29:05.2340502Z 
2023-01-11T21:29:05.2340609Z Running tests...
2023-01-11T21:29:05.2340987Z ----------------------------------------------------------------------
2023-01-11T21:29:05.2341584Z Test results will be stored in test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization
2023-01-11T21:29:05.2342129Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:29:05.2342612Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11699
2023-01-11T21:29:05.2343037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11700
2023-01-11T21:29:05.2343630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:29:05.2344069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:29:05.2344638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:29:05.2345085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:29:05.2345652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:29:05.2346088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:29:05.2346641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:29:05.2347100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:29:05.2347529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:29:05.2348010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:29:05.2348474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:29:05.2348952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:29:05.2349602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:29:05.2350110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:29:05.2350759Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:29:05.2351281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:29:05.2352578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:29:05.2353270Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:29:05.2353660Z ok (5.381s)
2023-01-11T21:29:05.2353808Z 
2023-01-11T21:29:05.2354072Z ----------------------------------------------------------------------
2023-01-11T21:29:05.2354395Z Ran 1 test in 5.382s
2023-01-11T21:29:05.2354536Z 
2023-01-11T21:29:05.2354628Z OK
2023-01-11T21:29:05.2354757Z 
2023-01-11T21:29:05.2354880Z Generating XML reports...
2023-01-11T21:29:05.2355629Z Generated XML report: test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212859.xml
2023-01-11T21:29:05.2356096Z 
2023-01-11T21:29:05.2356416Z ##[endgroup]
2023-01-11T21:29:05.2357126Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_cx85kh7g)
2023-01-11T21:29:05.2357560Z 
2023-01-11T21:29:05.2357688Z Shard 2: gloo should be run in 1
2023-01-11T21:29:05.2357947Z Shard 2: ucc should be run in 1
2023-01-11T21:29:05.2358367Z Running distributed/test_distributed_spawn ... [2023-01-11 21:29:05.227707]
2023-01-11T21:29:05.2358676Z /usr/bin/mpiexec
2023-01-11T21:29:05.2359049Z MPI not available -- MPI backend tests will be skipped
2023-01-11T21:29:05.2359579Z Map different backends to different shards for distributed/test_distributed_spawn: {'gloo': 1, 'nccl': 2, 'ucc': 3}
2023-01-11T21:29:05.2359968Z Shard 2: test should be run in 1
2023-01-11T21:29:05.2360336Z Running distributed tests for the nccl backend with env init_method in shard 2 of 3
2023-01-11T21:29:05.2361049Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:29:05.229328]
2023-01-11T21:53:36.8768819Z 
2023-01-11T21:53:36.8769371Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2023-01-11T21:53:36.8773280Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_qma1brgt)
2023-01-11T21:53:36.8776158Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:53:36.8852995Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward_grad_as_bucket_view_false>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward_ignored_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2023-01-11T21:53:36.8920653Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8927355Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8928173Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8928985Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8929801Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8930712Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8931656Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8932587Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8933503Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8934527Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8935635Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8936634Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8937638Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8938611Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8939499Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8940363Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8941241Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8942024Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8942810Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8943643Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8944571Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8945429Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8946188Z test_all_gather (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8946929Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8947709Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8948534Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8949429Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8950050Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8950456Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8950973Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8951415Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8951818Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8952751Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8953135Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8953557Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8953962Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8954345Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8954756Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8955165Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8955557Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8955961Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8956398Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8956838Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8957263Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8957685Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8958106Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8958520Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8958948Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8959359Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8959795Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8960219Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8960626Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8961031Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8961431Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8961841Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8962237Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8962645Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8963032Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8963421Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8963804Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8964178Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8964580Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8964946Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8965298Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8965676Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8966077Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8966475Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8966845Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8967211Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8967576Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8967943Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8968315Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8968841Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8969235Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8969615Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8969980Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8970351Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8970717Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8971107Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8971499Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8971870Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8972248Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8972645Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8973058Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8973485Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8973923Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8974369Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8974800Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8975238Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8975669Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8976081Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8976509Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8976944Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8977391Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8977826Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8978276Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8978718Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8979142Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8979561Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8979944Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8980312Z test_backend_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8980652Z test_barrier (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8981010Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8981390Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8981759Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8982143Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8982519Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8982894Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8983290Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8983681Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8984071Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8984451Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8984869Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8985277Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8985770Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8986185Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8986590Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8986996Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8987412Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8987818Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8988199Z test_broadcast (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8988548Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8988932Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8989314Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8989677Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8990070Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8990518Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8991030Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8991474Z test_ddp_apply_optim_in_backward (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8992467Z test_ddp_apply_optim_in_backward_grad_as_bucket_view_false (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8992962Z test_ddp_apply_optim_in_backward_ignored_params (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8993373Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8993784Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8994196Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8994616Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8995058Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8995517Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8995950Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8996355Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8996790Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8997190Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8997542Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8997928Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8998338Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8998752Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8999165Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.8999600Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9000010Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9000403Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9000861Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9001357Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9001917Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9002511Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9003268Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9003875Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9004477Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9005066Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9005659Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9006256Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9006804Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9007279Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9007715Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9008095Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9008471Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9008873Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9009264Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9009681Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9010100Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9010560Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9011029Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9011426Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9011811Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9012215Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9012645Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9013052Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9013460Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9013867Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9014275Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9014698Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9015119Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9015503Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9015906Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9016318Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9016710Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9017108Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9017563Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9017994Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9018370Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9018745Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9019239Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9019660Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9020052Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9020425Z test_gather (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9020783Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9021128Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9021499Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9021871Z test_gather_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9022271Z test_gather_object (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9022654Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9023030Z test_get_backend (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9023388Z test_get_future (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9023732Z test_get_rank (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9024104Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9024492Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9024859Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9025223Z test_irecv (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9025566Z test_isend (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9025922Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9026316Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9026723Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9027152Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9027596Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9028011Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9028424Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9028837Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9029261Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9029677Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9030068Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9030473Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9030878Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9031284Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9031657Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9032677Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9033154Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9033624Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9034090Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9034529Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9034986Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9035419Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9035845Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9036258Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9036662Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9037239Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9037686Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9038158Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9038647Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9039126Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9039540Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9039914Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9040311Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9040706Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9041076Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9041453Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9041834Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9042213Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9042560Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9042915Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9043283Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9043640Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9044026Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9044425Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9044777Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9045144Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9045530Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9045909Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9046251Z test_scatter (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9046607Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9046977Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9047328Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9047706Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9048098Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9048454Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9048831Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9049195Z test_send_recv (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9049564Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9049975Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9050419Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9050840Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9051212Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9051607Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9052028Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9052419Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9052810Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9053215Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9053643Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9054093Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9054543Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9054951Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9055327Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9055702Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9056114Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9056539Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9056971Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9057700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9058148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9058712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9059176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9059405Z 
2023-01-11T21:53:36.9059515Z Running tests...
2023-01-11T21:53:36.9059915Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9060423Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9061015Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9061575Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11852
2023-01-11T21:53:36.9062004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11853
2023-01-11T21:53:36.9062601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9063051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9063618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9064065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9064636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9065072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9065615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9066071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9066518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9067029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9067672Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9068357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9068877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9069345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9069856Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T21:53:36.9070695Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T21:53:36.9071470Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T21:53:36.9072572Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T21:53:36.9073225Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T21:53:36.9074043Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T21:53:36.9074700Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T21:53:36.9075519Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T21:53:36.9076179Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T21:53:36.9076997Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T21:53:36.9077657Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T21:53:36.9078471Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T21:53:36.9079118Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T21:53:36.9079932Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T21:53:36.9080601Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T21:53:36.9081422Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T21:53:36.9081893Z ok (5.760s)
2023-01-11T21:53:36.9082041Z 
2023-01-11T21:53:36.9082308Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9082630Z Ran 1 test in 5.760s
2023-01-11T21:53:36.9082787Z 
2023-01-11T21:53:36.9082873Z OK
2023-01-11T21:53:36.9082986Z 
2023-01-11T21:53:36.9083103Z Generating XML reports...
2023-01-11T21:53:36.9083698Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212909.xml
2023-01-11T21:53:36.9084404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9084842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9085408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9085865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9086091Z 
2023-01-11T21:53:36.9086193Z Running tests...
2023-01-11T21:53:36.9086569Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9087090Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9087621Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2023-01-11T21:53:36.9087929Z 
2023-01-11T21:53:36.9088172Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9088490Z Ran 1 test in 0.003s
2023-01-11T21:53:36.9088646Z 
2023-01-11T21:53:36.9088753Z OK (skipped=1)
2023-01-11T21:53:36.9088996Z 
2023-01-11T21:53:36.9089165Z Generating XML reports...
2023-01-11T21:53:36.9089761Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212917.xml
2023-01-11T21:53:36.9090467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9090911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9091461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9091927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9092157Z 
2023-01-11T21:53:36.9092263Z Running tests...
2023-01-11T21:53:36.9092658Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9093164Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9093676Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9094166Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12000
2023-01-11T21:53:36.9094616Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12001
2023-01-11T21:53:36.9095202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9095647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9096214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9096661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9097228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9097667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9098234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9098675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9099124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9099627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9100265Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9100951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9101469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9101944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9102273Z ok (4.280s)
2023-01-11T21:53:36.9102417Z 
2023-01-11T21:53:36.9102680Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9102999Z Ran 1 test in 4.281s
2023-01-11T21:53:36.9103157Z 
2023-01-11T21:53:36.9103248Z OK
2023-01-11T21:53:36.9103362Z 
2023-01-11T21:53:36.9103481Z Generating XML reports...
2023-01-11T21:53:36.9104078Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212920.xml
2023-01-11T21:53:36.9104781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9105211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9105775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9106340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9106571Z 
2023-01-11T21:53:36.9106678Z Running tests...
2023-01-11T21:53:36.9107055Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9107576Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9108104Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9109139Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.629s)
2023-01-11T21:53:36.9109676Z 
2023-01-11T21:53:36.9109920Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9110241Z Ran 1 test in 1.629s
2023-01-11T21:53:36.9110405Z 
2023-01-11T21:53:36.9110513Z OK (skipped=1)
2023-01-11T21:53:36.9110665Z 
2023-01-11T21:53:36.9110787Z Generating XML reports...
2023-01-11T21:53:36.9111367Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212926.xml
2023-01-11T21:53:36.9112307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9112758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9113317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9113784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9114013Z 
2023-01-11T21:53:36.9114123Z Running tests...
2023-01-11T21:53:36.9114520Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9115028Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9115594Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2023-01-11T21:53:36.9115922Z 
2023-01-11T21:53:36.9116178Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9116497Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9116639Z 
2023-01-11T21:53:36.9116744Z OK (skipped=1)
2023-01-11T21:53:36.9116895Z 
2023-01-11T21:53:36.9117016Z Generating XML reports...
2023-01-11T21:53:36.9117613Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212930.xml
2023-01-11T21:53:36.9118305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9118744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9119317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9119786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9119997Z 
2023-01-11T21:53:36.9120103Z Running tests...
2023-01-11T21:53:36.9120498Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9121018Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9121568Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2023-01-11T21:53:36.9121915Z 
2023-01-11T21:53:36.9122226Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9122549Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9122705Z 
2023-01-11T21:53:36.9122812Z OK (skipped=1)
2023-01-11T21:53:36.9122945Z 
2023-01-11T21:53:36.9123067Z Generating XML reports...
2023-01-11T21:53:36.9123828Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212933.xml
2023-01-11T21:53:36.9124546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9124976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9125545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9126010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9126238Z 
2023-01-11T21:53:36.9126345Z Running tests...
2023-01-11T21:53:36.9126727Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9127249Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9127800Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9128345Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12205
2023-01-11T21:53:36.9128776Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12206
2023-01-11T21:53:36.9129371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9129813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9130364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9130822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9131393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9131825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9132376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9132837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9133287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9133768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9134427Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9135112Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9135633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9136087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9136567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9137049Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9137531Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9137989Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9138465Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9138934Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9139391Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9139865Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9140336Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9140915Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9141383Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9141855Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9142197Z ok (6.718s)
2023-01-11T21:53:36.9142343Z 
2023-01-11T21:53:36.9142595Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9142916Z Ran 1 test in 6.718s
2023-01-11T21:53:36.9143073Z 
2023-01-11T21:53:36.9143164Z OK
2023-01-11T21:53:36.9143294Z 
2023-01-11T21:53:36.9143415Z Generating XML reports...
2023-01-11T21:53:36.9143999Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212935.xml
2023-01-11T21:53:36.9144708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9145160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9145709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9146173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9146403Z 
2023-01-11T21:53:36.9146509Z Running tests...
2023-01-11T21:53:36.9146905Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9147414Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9147982Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9148524Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12324
2023-01-11T21:53:36.9148954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12325
2023-01-11T21:53:36.9149570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9150012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9150582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9151030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9151600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9152275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9152853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9153298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9153752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9154256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9154895Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9155579Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9156098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9156564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9157026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9157509Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9158129Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9158616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9158946Z ok (5.595s)
2023-01-11T21:53:36.9159091Z 
2023-01-11T21:53:36.9159364Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9159687Z Ran 1 test in 5.595s
2023-01-11T21:53:36.9159828Z 
2023-01-11T21:53:36.9159920Z OK
2023-01-11T21:53:36.9160050Z 
2023-01-11T21:53:36.9160173Z Generating XML reports...
2023-01-11T21:53:36.9160775Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212944.xml
2023-01-11T21:53:36.9161464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9161910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9162478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9162945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9163170Z 
2023-01-11T21:53:36.9163260Z Running tests...
2023-01-11T21:53:36.9163653Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9164173Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9164745Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9165274Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12443
2023-01-11T21:53:36.9165723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12444
2023-01-11T21:53:36.9166324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9166761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9167328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9167789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9168357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9168774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9169333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9169787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9170222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9170721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9171384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9172075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9172577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9173044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9173518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9173999Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9174463Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9174944Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9175530Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9175991Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9176471Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9176942Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9177280Z ok (5.830s)
2023-01-11T21:53:36.9177408Z 
2023-01-11T21:53:36.9177678Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9177999Z Ran 1 test in 5.830s
2023-01-11T21:53:36.9178157Z 
2023-01-11T21:53:36.9178248Z OK
2023-01-11T21:53:36.9178378Z 
2023-01-11T21:53:36.9178482Z Generating XML reports...
2023-01-11T21:53:36.9179084Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212953.xml
2023-01-11T21:53:36.9179799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9180245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9180795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9181256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9181484Z 
2023-01-11T21:53:36.9181592Z Running tests...
2023-01-11T21:53:36.9181970Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9182492Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9183086Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9183660Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12562
2023-01-11T21:53:36.9184090Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12563
2023-01-11T21:53:36.9184680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9185125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9185691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9186138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9186709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9187149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9187695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9188161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9188610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9189107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9189747Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9190438Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9190961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9191431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9192115Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9192754Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9193111Z ok (5.795s)
2023-01-11T21:53:36.9193256Z 
2023-01-11T21:53:36.9193516Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9193837Z Ran 1 test in 5.795s
2023-01-11T21:53:36.9193996Z 
2023-01-11T21:53:36.9194087Z OK
2023-01-11T21:53:36.9194218Z 
2023-01-11T21:53:36.9194322Z Generating XML reports...
2023-01-11T21:53:36.9194924Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213001.xml
2023-01-11T21:53:36.9195630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9196080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9196632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9197253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9197672Z 
2023-01-11T21:53:36.9197789Z Running tests...
2023-01-11T21:53:36.9198201Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9198708Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9199297Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9199862Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12681
2023-01-11T21:53:36.9200296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12682
2023-01-11T21:53:36.9200894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9201339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9201916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9202368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9202933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9203367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9203916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9204374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9204822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9205320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9205965Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9206655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9207173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9207642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9208106Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9208587Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9209068Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9209529Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9209957Z ok (6.265s)
2023-01-11T21:53:36.9210142Z 
2023-01-11T21:53:36.9210413Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9210737Z Ran 1 test in 6.266s
2023-01-11T21:53:36.9210879Z 
2023-01-11T21:53:36.9210970Z OK
2023-01-11T21:53:36.9211100Z 
2023-01-11T21:53:36.9211221Z Generating XML reports...
2023-01-11T21:53:36.9211823Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213009.xml
2023-01-11T21:53:36.9212519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9212962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9213532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9213995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9214208Z 
2023-01-11T21:53:36.9214318Z Running tests...
2023-01-11T21:53:36.9214712Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9215234Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9215783Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9216328Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12800
2023-01-11T21:53:36.9216773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12801
2023-01-11T21:53:36.9217372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9217799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9218364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9218831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9219399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9219818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9220375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9220831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9221261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9221756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9222458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9223155Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9223659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9224127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9224605Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9225128Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9225592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9226064Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9226406Z ok (6.097s)
2023-01-11T21:53:36.9226551Z 
2023-01-11T21:53:36.9226801Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9227185Z Ran 1 test in 6.097s
2023-01-11T21:53:36.9227383Z 
2023-01-11T21:53:36.9227477Z OK
2023-01-11T21:53:36.9227608Z 
2023-01-11T21:53:36.9227714Z Generating XML reports...
2023-01-11T21:53:36.9228320Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213018.xml
2023-01-11T21:53:36.9229032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9229667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9230359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9230824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9231052Z 
2023-01-11T21:53:36.9231160Z Running tests...
2023-01-11T21:53:36.9231537Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9232331Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9232931Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9233497Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12919
2023-01-11T21:53:36.9233926Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12920
2023-01-11T21:53:36.9234529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9234975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9235542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9235990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9236567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9237008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9237556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9238017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9238464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9238958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9239596Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9240282Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9240807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9241276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9241737Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9242220Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9242696Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9243158Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9243504Z ok (5.681s)
2023-01-11T21:53:36.9243651Z 
2023-01-11T21:53:36.9243918Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9244237Z Ran 1 test in 5.681s
2023-01-11T21:53:36.9244377Z 
2023-01-11T21:53:36.9244467Z OK
2023-01-11T21:53:36.9244700Z 
2023-01-11T21:53:36.9244872Z Generating XML reports...
2023-01-11T21:53:36.9245485Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213027.xml
2023-01-11T21:53:36.9246173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9246620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9247188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9247651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9247860Z 
2023-01-11T21:53:36.9247968Z Running tests...
2023-01-11T21:53:36.9248360Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9248884Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9249429Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9250498Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.674s)
2023-01-11T21:53:36.9251019Z 
2023-01-11T21:53:36.9251277Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9251596Z Ran 1 test in 1.674s
2023-01-11T21:53:36.9251755Z 
2023-01-11T21:53:36.9251860Z OK (skipped=1)
2023-01-11T21:53:36.9251997Z 
2023-01-11T21:53:36.9252119Z Generating XML reports...
2023-01-11T21:53:36.9252715Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213035.xml
2023-01-11T21:53:36.9253426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9253859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9254426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9254887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9255112Z 
2023-01-11T21:53:36.9255220Z Running tests...
2023-01-11T21:53:36.9255597Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9256125Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9256675Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9257186Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13072
2023-01-11T21:53:36.9257639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13073
2023-01-11T21:53:36.9258240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9258683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9259232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9259690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9260258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9260678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9261241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9261695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9262241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9262729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9263387Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9264073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9264588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9265038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9265371Z ok (4.375s)
2023-01-11T21:53:36.9265517Z 
2023-01-11T21:53:36.9265780Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9266087Z Ran 1 test in 4.375s
2023-01-11T21:53:36.9266245Z 
2023-01-11T21:53:36.9266339Z OK
2023-01-11T21:53:36.9266471Z 
2023-01-11T21:53:36.9266592Z Generating XML reports...
2023-01-11T21:53:36.9267192Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213039.xml
2023-01-11T21:53:36.9267884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9268326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9268892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9269339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9269567Z 
2023-01-11T21:53:36.9269674Z Running tests...
2023-01-11T21:53:36.9270068Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9270599Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9271147Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9272452Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.670s)
2023-01-11T21:53:36.9272981Z 
2023-01-11T21:53:36.9273251Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9273572Z Ran 1 test in 1.671s
2023-01-11T21:53:36.9273730Z 
2023-01-11T21:53:36.9273819Z OK (skipped=1)
2023-01-11T21:53:36.9273968Z 
2023-01-11T21:53:36.9274091Z Generating XML reports...
2023-01-11T21:53:36.9274696Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213046.xml
2023-01-11T21:53:36.9275407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9275833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9276399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9276863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9277089Z 
2023-01-11T21:53:36.9277179Z Running tests...
2023-01-11T21:53:36.9277571Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9278096Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9278627Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9279263Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13211
2023-01-11T21:53:36.9279714Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13212
2023-01-11T21:53:36.9280319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9280748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9281312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9281775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9282344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9282766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9283329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9283794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9284227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9284723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9285382Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9286068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9286570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9287047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9287389Z ok (5.807s)
2023-01-11T21:53:36.9287535Z 
2023-01-11T21:53:36.9287808Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9288114Z Ran 1 test in 5.807s
2023-01-11T21:53:36.9288271Z 
2023-01-11T21:53:36.9288363Z OK
2023-01-11T21:53:36.9288492Z 
2023-01-11T21:53:36.9288613Z Generating XML reports...
2023-01-11T21:53:36.9289198Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213050.xml
2023-01-11T21:53:36.9289907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9290350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9290916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9291364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9291590Z 
2023-01-11T21:53:36.9291696Z Running tests...
2023-01-11T21:53:36.9292100Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9292607Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9293104Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2023-01-11T21:53:36.9293388Z 
2023-01-11T21:53:36.9293649Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9293968Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9294108Z 
2023-01-11T21:53:36.9294212Z OK (skipped=1)
2023-01-11T21:53:36.9294363Z 
2023-01-11T21:53:36.9294483Z Generating XML reports...
2023-01-11T21:53:36.9295078Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213059.xml
2023-01-11T21:53:36.9295764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9296206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9296887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9297356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9297585Z 
2023-01-11T21:53:36.9297675Z Running tests...
2023-01-11T21:53:36.9298068Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9298589Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9299025Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9299503Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9299984Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13359
2023-01-11T21:53:36.9300428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13360
2023-01-11T21:53:36.9301022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9301469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9302035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9302500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9303054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9303495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9304053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9304498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9304955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9305449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9306105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9306779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9307299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9307772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9308096Z ok (6.232s)
2023-01-11T21:53:36.9308245Z 
2023-01-11T21:53:36.9308510Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9308831Z Ran 1 test in 6.232s
2023-01-11T21:53:36.9308995Z 
2023-01-11T21:53:36.9309086Z OK
2023-01-11T21:53:36.9309201Z 
2023-01-11T21:53:36.9309325Z Generating XML reports...
2023-01-11T21:53:36.9309925Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213101.xml
2023-01-11T21:53:36.9310633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9311063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9311629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9312322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9312555Z 
2023-01-11T21:53:36.9312663Z Running tests...
2023-01-11T21:53:36.9313055Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9313579Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9314193Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9314702Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9315171Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13478
2023-01-11T21:53:36.9315615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13479
2023-01-11T21:53:36.9316217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9316646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9317215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9317677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9318250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9318674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9319239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9319701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9320136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9320630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9321287Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9321973Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9322535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9323010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9323490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9323970Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9324302Z ok (6.142s)
2023-01-11T21:53:36.9324447Z 
2023-01-11T21:53:36.9324714Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9325036Z Ran 1 test in 6.142s
2023-01-11T21:53:36.9325194Z 
2023-01-11T21:53:36.9325267Z OK
2023-01-11T21:53:36.9325399Z 
2023-01-11T21:53:36.9325519Z Generating XML reports...
2023-01-11T21:53:36.9326119Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213110.xml
2023-01-11T21:53:36.9326838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9327267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9327837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9328299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9328525Z 
2023-01-11T21:53:36.9328615Z Running tests...
2023-01-11T21:53:36.9329008Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9329528Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9330017Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9330535Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9331133Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13597
2023-01-11T21:53:36.9331590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13598
2023-01-11T21:53:36.9332174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9332621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9333188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9333647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9334203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9334643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9335214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9335659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9336106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9336603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9337256Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9337925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9338444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9338920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9339398Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9339869Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9340218Z ok (6.302s)
2023-01-11T21:53:36.9340368Z 
2023-01-11T21:53:36.9340630Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9340937Z Ran 1 test in 6.302s
2023-01-11T21:53:36.9341095Z 
2023-01-11T21:53:36.9341186Z OK
2023-01-11T21:53:36.9341316Z 
2023-01-11T21:53:36.9341443Z Generating XML reports...
2023-01-11T21:53:36.9342052Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213119.xml
2023-01-11T21:53:36.9342749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9343194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9343771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9344226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9344455Z 
2023-01-11T21:53:36.9344561Z Running tests...
2023-01-11T21:53:36.9344958Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9345482Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9345937Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:53:36.9346423Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9346900Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13716
2023-01-11T21:53:36.9347328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13717
2023-01-11T21:53:36.9347932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9348478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9349054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9349499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9350069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9350507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9351072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9351514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9352185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9352700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9353346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9354032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9354553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9355020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9355345Z ok (6.159s)
2023-01-11T21:53:36.9355490Z 
2023-01-11T21:53:36.9355755Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9356078Z Ran 1 test in 6.159s
2023-01-11T21:53:36.9356235Z 
2023-01-11T21:53:36.9356309Z OK
2023-01-11T21:53:36.9356440Z 
2023-01-11T21:53:36.9356566Z Generating XML reports...
2023-01-11T21:53:36.9357172Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213127.xml
2023-01-11T21:53:36.9357878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9358309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9358877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9359337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9359562Z 
2023-01-11T21:53:36.9359670Z Running tests...
2023-01-11T21:53:36.9360050Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9360571Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9361068Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:36.9361354Z 
2023-01-11T21:53:36.9361599Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9361918Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9362078Z 
2023-01-11T21:53:36.9362184Z OK (skipped=1)
2023-01-11T21:53:36.9362333Z 
2023-01-11T21:53:36.9362453Z Generating XML reports...
2023-01-11T21:53:36.9363028Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213136.xml
2023-01-11T21:53:36.9363726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9364171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9364716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9365174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9365488Z 
2023-01-11T21:53:36.9365642Z Running tests...
2023-01-11T21:53:36.9366051Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9366560Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9367102Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2023-01-11T21:53:36.9367419Z 
2023-01-11T21:53:36.9367680Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9367981Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9368137Z 
2023-01-11T21:53:36.9368242Z OK (skipped=1)
2023-01-11T21:53:36.9368392Z 
2023-01-11T21:53:36.9368514Z Generating XML reports...
2023-01-11T21:53:36.9369108Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213139.xml
2023-01-11T21:53:36.9369804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9370245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9370816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9371263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9371489Z 
2023-01-11T21:53:36.9371594Z Running tests...
2023-01-11T21:53:36.9371985Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9372508Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9373034Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2023-01-11T21:53:36.9373352Z 
2023-01-11T21:53:36.9373610Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9373937Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9374093Z 
2023-01-11T21:53:36.9374198Z OK (skipped=1)
2023-01-11T21:53:36.9374331Z 
2023-01-11T21:53:36.9374451Z Generating XML reports...
2023-01-11T21:53:36.9375048Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213141.xml
2023-01-11T21:53:36.9375747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9376176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9376744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9377206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9377430Z 
2023-01-11T21:53:36.9377535Z Running tests...
2023-01-11T21:53:36.9377917Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9378445Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9378980Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2023-01-11T21:53:36.9379293Z 
2023-01-11T21:53:36.9379537Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9379856Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9380012Z 
2023-01-11T21:53:36.9380116Z OK (skipped=1)
2023-01-11T21:53:36.9380268Z 
2023-01-11T21:53:36.9380388Z Generating XML reports...
2023-01-11T21:53:36.9380968Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213143.xml
2023-01-11T21:53:36.9381673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9382120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9382769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9383243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9383473Z 
2023-01-11T21:53:36.9383579Z Running tests...
2023-01-11T21:53:36.9383976Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9384482Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9385022Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2023-01-11T21:53:36.9385336Z 
2023-01-11T21:53:36.9385598Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9385916Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9386054Z 
2023-01-11T21:53:36.9386159Z OK (skipped=1)
2023-01-11T21:53:36.9386308Z 
2023-01-11T21:53:36.9386434Z Generating XML reports...
2023-01-11T21:53:36.9387035Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213146.xml
2023-01-11T21:53:36.9387727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9388176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9388743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9389204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9389414Z 
2023-01-11T21:53:36.9389520Z Running tests...
2023-01-11T21:53:36.9389911Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9390432Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9390960Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2023-01-11T21:53:36.9391285Z 
2023-01-11T21:53:36.9391545Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9392214Z Ran 1 test in 0.003s
2023-01-11T21:53:36.9392385Z 
2023-01-11T21:53:36.9392492Z OK (skipped=1)
2023-01-11T21:53:36.9392628Z 
2023-01-11T21:53:36.9392751Z Generating XML reports...
2023-01-11T21:53:36.9393360Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213148.xml
2023-01-11T21:53:36.9394070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9394515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9395062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9395536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9395762Z 
2023-01-11T21:53:36.9395870Z Running tests...
2023-01-11T21:53:36.9396250Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9396777Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9397289Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:36.9397583Z 
2023-01-11T21:53:36.9397840Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9398142Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9398300Z 
2023-01-11T21:53:36.9398406Z OK (skipped=1)
2023-01-11T21:53:36.9398556Z 
2023-01-11T21:53:36.9398676Z Generating XML reports...
2023-01-11T21:53:36.9399255Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213151.xml
2023-01-11T21:53:36.9400108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9400563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9401132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9401581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9401809Z 
2023-01-11T21:53:36.9401915Z Running tests...
2023-01-11T21:53:36.9402308Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9402819Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9403321Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9403809Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14066
2023-01-11T21:53:36.9404264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14067
2023-01-11T21:53:36.9404849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9405296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9405865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9406312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9406882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9407321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9407884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9408327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9408785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9409278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9409932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9410607Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9411127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9411592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9412157Z STAGE:2023-01-11 21:31:58 14066:14066 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9412750Z STAGE:2023-01-11 21:31:58 14067:14067 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9413330Z STAGE:2023-01-11 21:31:59 14066:14066 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9413910Z STAGE:2023-01-11 21:31:59 14067:14067 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9414484Z STAGE:2023-01-11 21:31:59 14066:14066 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9415087Z STAGE:2023-01-11 21:31:59 14067:14067 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9415658Z STAGE:2023-01-11 21:31:59 14067:14067 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9416224Z STAGE:2023-01-11 21:31:59 14066:14066 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9416784Z STAGE:2023-01-11 21:31:59 14067:14067 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9417488Z STAGE:2023-01-11 21:31:59 14067:14067 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9418084Z STAGE:2023-01-11 21:31:59 14066:14066 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9418668Z STAGE:2023-01-11 21:31:59 14066:14066 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9419002Z ok (6.490s)
2023-01-11T21:53:36.9419149Z 
2023-01-11T21:53:36.9419410Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9419729Z Ran 1 test in 6.490s
2023-01-11T21:53:36.9419886Z 
2023-01-11T21:53:36.9419960Z OK
2023-01-11T21:53:36.9420088Z 
2023-01-11T21:53:36.9420209Z Generating XML reports...
2023-01-11T21:53:36.9420660Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213153.xml
2023-01-11T21:53:36.9421032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9421213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9421590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9421765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9421801Z 
2023-01-11T21:53:36.9421891Z Running tests...
2023-01-11T21:53:36.9422194Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9422515Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9422786Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9423003Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14185
2023-01-11T21:53:36.9423216Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14186
2023-01-11T21:53:36.9423593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9423766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9424127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9424314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9424678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9424850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9425224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9425415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9425663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9425911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9426294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9426693Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9426922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9427149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9427485Z STAGE:2023-01-11 21:32:07 14186:14186 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9427817Z STAGE:2023-01-11 21:32:07 14185:14185 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9428251Z STAGE:2023-01-11 21:32:08 14185:14185 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9428593Z STAGE:2023-01-11 21:32:08 14186:14186 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9428939Z STAGE:2023-01-11 21:32:08 14185:14185 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9429268Z STAGE:2023-01-11 21:32:08 14186:14186 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9429596Z STAGE:2023-01-11 21:32:08 14186:14186 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9429925Z STAGE:2023-01-11 21:32:08 14186:14186 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9430266Z STAGE:2023-01-11 21:32:08 14186:14186 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9430591Z STAGE:2023-01-11 21:32:08 14185:14185 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9430929Z STAGE:2023-01-11 21:32:08 14185:14185 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9431272Z STAGE:2023-01-11 21:32:08 14185:14185 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9431372Z ok (6.507s)
2023-01-11T21:53:36.9431392Z 
2023-01-11T21:53:36.9431655Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9431748Z Ran 1 test in 6.507s
2023-01-11T21:53:36.9431767Z 
2023-01-11T21:53:36.9432083Z OK
2023-01-11T21:53:36.9432105Z 
2023-01-11T21:53:36.9432242Z Generating XML reports...
2023-01-11T21:53:36.9432704Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213202.xml
2023-01-11T21:53:36.9433070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9433242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9433627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9433817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9433837Z 
2023-01-11T21:53:36.9433943Z Running tests...
2023-01-11T21:53:36.9434185Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9434497Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9434766Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:36.9434786Z 
2023-01-11T21:53:36.9435043Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9435152Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9435170Z 
2023-01-11T21:53:36.9435275Z OK (skipped=1)
2023-01-11T21:53:36.9435294Z 
2023-01-11T21:53:36.9435419Z Generating XML reports...
2023-01-11T21:53:36.9435868Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213211.xml
2023-01-11T21:53:36.9436220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9436394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9436768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9436957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9436977Z 
2023-01-11T21:53:36.9437083Z Running tests...
2023-01-11T21:53:36.9437340Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9437652Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9437988Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:36.9438058Z 
2023-01-11T21:53:36.9438329Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9438423Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9438442Z 
2023-01-11T21:53:36.9438547Z OK (skipped=1)
2023-01-11T21:53:36.9438566Z 
2023-01-11T21:53:36.9438684Z Generating XML reports...
2023-01-11T21:53:36.9439127Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213213.xml
2023-01-11T21:53:36.9439496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9439668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9440044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9440237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9440260Z 
2023-01-11T21:53:36.9440366Z Running tests...
2023-01-11T21:53:36.9440607Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9440919Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9441197Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9441416Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14370
2023-01-11T21:53:36.9441628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14371
2023-01-11T21:53:36.9441993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9442165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9442543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9442718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9443084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9443255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9443632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9443820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9444068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9444306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9444704Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9445106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9445320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9445545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9445883Z STAGE:2023-01-11 21:32:21 14371:14371 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9446215Z STAGE:2023-01-11 21:32:21 14370:14370 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9446549Z STAGE:2023-01-11 21:32:21 14370:14370 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9446881Z STAGE:2023-01-11 21:32:21 14371:14371 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9447284Z STAGE:2023-01-11 21:32:21 14370:14370 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9447678Z STAGE:2023-01-11 21:32:21 14371:14371 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9447780Z ok (6.396s)
2023-01-11T21:53:36.9447800Z 
2023-01-11T21:53:36.9448043Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9448151Z Ran 1 test in 6.397s
2023-01-11T21:53:36.9448170Z 
2023-01-11T21:53:36.9448260Z OK
2023-01-11T21:53:36.9448279Z 
2023-01-11T21:53:36.9448400Z Generating XML reports...
2023-01-11T21:53:36.9448848Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213216.xml
2023-01-11T21:53:36.9449216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9449388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9449767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9449944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9449980Z 
2023-01-11T21:53:36.9450070Z Running tests...
2023-01-11T21:53:36.9450330Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9450641Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9450921Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9451138Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14489
2023-01-11T21:53:36.9451351Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14490
2023-01-11T21:53:36.9451716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9451894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9452253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9452442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9452806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9452977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9453349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9453534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9453780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9454017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9454404Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9454799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9455030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9455256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9455588Z STAGE:2023-01-11 21:32:30 14490:14490 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9455918Z STAGE:2023-01-11 21:32:30 14489:14489 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9456250Z STAGE:2023-01-11 21:32:30 14489:14489 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9456686Z STAGE:2023-01-11 21:32:30 14490:14490 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9457039Z STAGE:2023-01-11 21:32:30 14489:14489 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9457367Z STAGE:2023-01-11 21:32:30 14490:14490 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9457467Z ok (6.282s)
2023-01-11T21:53:36.9457487Z 
2023-01-11T21:53:36.9457744Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9457855Z Ran 1 test in 6.282s
2023-01-11T21:53:36.9457874Z 
2023-01-11T21:53:36.9457964Z OK
2023-01-11T21:53:36.9457983Z 
2023-01-11T21:53:36.9458103Z Generating XML reports...
2023-01-11T21:53:36.9458550Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213225.xml
2023-01-11T21:53:36.9458917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9459097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9459459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9459647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9459666Z 
2023-01-11T21:53:36.9459774Z Running tests...
2023-01-11T21:53:36.9460030Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9460338Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9460604Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9461354Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81029 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.642s)
2023-01-11T21:53:36.9461378Z 
2023-01-11T21:53:36.9461636Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9461744Z Ran 1 test in 1.642s
2023-01-11T21:53:36.9461763Z 
2023-01-11T21:53:36.9461850Z OK (skipped=1)
2023-01-11T21:53:36.9461885Z 
2023-01-11T21:53:36.9461991Z Generating XML reports...
2023-01-11T21:53:36.9462438Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213234.xml
2023-01-11T21:53:36.9462804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9462976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9463351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9463544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9463566Z 
2023-01-11T21:53:36.9463673Z Running tests...
2023-01-11T21:53:36.9463933Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9464228Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9464502Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9465244Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81023 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.647s)
2023-01-11T21:53:36.9465264Z 
2023-01-11T21:53:36.9465519Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9465627Z Ran 1 test in 1.647s
2023-01-11T21:53:36.9465696Z 
2023-01-11T21:53:36.9465805Z OK (skipped=1)
2023-01-11T21:53:36.9465861Z 
2023-01-11T21:53:36.9465986Z Generating XML reports...
2023-01-11T21:53:36.9466434Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213238.xml
2023-01-11T21:53:36.9466802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9466974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9467335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9467523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9467542Z 
2023-01-11T21:53:36.9467649Z Running tests...
2023-01-11T21:53:36.9467909Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9468219Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9468503Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9468723Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14676
2023-01-11T21:53:36.9468934Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14677
2023-01-11T21:53:36.9469283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9469456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9469829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9470019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9470381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9470559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9470933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9471121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9471367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9471589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9472219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9472628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9472857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9473090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9473190Z ok (5.791s)
2023-01-11T21:53:36.9473211Z 
2023-01-11T21:53:36.9473472Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9473583Z Ran 1 test in 5.791s
2023-01-11T21:53:36.9473602Z 
2023-01-11T21:53:36.9473674Z OK
2023-01-11T21:53:36.9473710Z 
2023-01-11T21:53:36.9473815Z Generating XML reports...
2023-01-11T21:53:36.9474260Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213242.xml
2023-01-11T21:53:36.9474627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9474800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9475173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9475496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9475517Z 
2023-01-11T21:53:36.9475627Z Running tests...
2023-01-11T21:53:36.9475888Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9476185Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9476458Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9476677Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14791
2023-01-11T21:53:36.9476889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14792
2023-01-11T21:53:36.9477257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9477427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9477811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9478000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9478365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9478522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9478895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9479082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9479327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9479564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9479970Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9480364Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9480591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9480801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9481045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:53:36.9481280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:53:36.9481675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:36.9482067Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:36.9482313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:53:36.9482548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:53:36.9482938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:53:36.9483331Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:53:36.9483573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T21:53:36.9483789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T21:53:36.9484182Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2023-01-11T21:53:36.9484674Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2023-01-11T21:53:36.9484780Z ok (6.582s)
2023-01-11T21:53:36.9484800Z 
2023-01-11T21:53:36.9485062Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9485172Z Ran 1 test in 6.582s
2023-01-11T21:53:36.9485191Z 
2023-01-11T21:53:36.9485281Z OK
2023-01-11T21:53:36.9485300Z 
2023-01-11T21:53:36.9485422Z Generating XML reports...
2023-01-11T21:53:36.9485854Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213250.xml
2023-01-11T21:53:36.9486225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9486399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9486781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9486974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9486994Z 
2023-01-11T21:53:36.9487100Z Running tests...
2023-01-11T21:53:36.9487358Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9487670Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9487932Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9488134Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14942
2023-01-11T21:53:36.9488347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14943
2023-01-11T21:53:36.9488713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9488885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9489264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9489453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9489811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9489982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9490341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9490532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9490775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9491013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9491420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9491813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9492043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9492269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9492368Z ok (5.795s)
2023-01-11T21:53:36.9492388Z 
2023-01-11T21:53:36.9492631Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9492739Z Ran 1 test in 5.795s
2023-01-11T21:53:36.9492758Z 
2023-01-11T21:53:36.9492849Z OK
2023-01-11T21:53:36.9492868Z 
2023-01-11T21:53:36.9492989Z Generating XML reports...
2023-01-11T21:53:36.9493435Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213259.xml
2023-01-11T21:53:36.9493902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9494082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9494458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9494648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9494668Z 
2023-01-11T21:53:36.9494758Z Running tests...
2023-01-11T21:53:36.9495017Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9495330Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9495746Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:53:36.9495769Z 
2023-01-11T21:53:36.9496028Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9496137Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9496156Z 
2023-01-11T21:53:36.9496261Z OK (skipped=1)
2023-01-11T21:53:36.9496279Z 
2023-01-11T21:53:36.9496400Z Generating XML reports...
2023-01-11T21:53:36.9496830Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213308.xml
2023-01-11T21:53:36.9497197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9497368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9497744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9497931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9497950Z 
2023-01-11T21:53:36.9498061Z Running tests...
2023-01-11T21:53:36.9498321Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9498634Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9499050Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:53:36.9499070Z 
2023-01-11T21:53:36.9499307Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9499417Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9499436Z 
2023-01-11T21:53:36.9499540Z OK (skipped=1)
2023-01-11T21:53:36.9499559Z 
2023-01-11T21:53:36.9499680Z Generating XML reports...
2023-01-11T21:53:36.9500124Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213310.xml
2023-01-11T21:53:36.9500488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9500666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9501044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9501232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9501251Z 
2023-01-11T21:53:36.9501340Z Running tests...
2023-01-11T21:53:36.9501597Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9501908Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9502336Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:53:36.9502356Z 
2023-01-11T21:53:36.9502614Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9502722Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9502794Z 
2023-01-11T21:53:36.9502902Z OK (skipped=1)
2023-01-11T21:53:36.9502959Z 
2023-01-11T21:53:36.9503085Z Generating XML reports...
2023-01-11T21:53:36.9503534Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213312.xml
2023-01-11T21:53:36.9503884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9504054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9504430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9504616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9504636Z 
2023-01-11T21:53:36.9504741Z Running tests...
2023-01-11T21:53:36.9505000Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9505310Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9505734Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:53:36.9505754Z 
2023-01-11T21:53:36.9506009Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9506101Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9506120Z 
2023-01-11T21:53:36.9506224Z OK (skipped=1)
2023-01-11T21:53:36.9506243Z 
2023-01-11T21:53:36.9506363Z Generating XML reports...
2023-01-11T21:53:36.9506807Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213315.xml
2023-01-11T21:53:36.9507171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9507343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9507720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9507913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9507932Z 
2023-01-11T21:53:36.9508037Z Running tests...
2023-01-11T21:53:36.9508280Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9508590Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9509000Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:53:36.9509019Z 
2023-01-11T21:53:36.9509275Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9509383Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9509402Z 
2023-01-11T21:53:36.9509505Z OK (skipped=1)
2023-01-11T21:53:36.9509524Z 
2023-01-11T21:53:36.9509644Z Generating XML reports...
2023-01-11T21:53:36.9510092Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213317.xml
2023-01-11T21:53:36.9510441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9510613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9510988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9511176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9511195Z 
2023-01-11T21:53:36.9511300Z Running tests...
2023-01-11T21:53:36.9511592Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9512393Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9513003Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:53:36.9513165Z 
2023-01-11T21:53:36.9513441Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9513535Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9513553Z 
2023-01-11T21:53:36.9513661Z OK (skipped=1)
2023-01-11T21:53:36.9513679Z 
2023-01-11T21:53:36.9513799Z Generating XML reports...
2023-01-11T21:53:36.9514248Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213320.xml
2023-01-11T21:53:36.9514618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9514790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9515166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9515354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9515378Z 
2023-01-11T21:53:36.9515488Z Running tests...
2023-01-11T21:53:36.9515731Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9516042Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9516458Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:53:36.9516478Z 
2023-01-11T21:53:36.9516733Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9516842Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9516861Z 
2023-01-11T21:53:36.9516966Z OK (skipped=1)
2023-01-11T21:53:36.9516984Z 
2023-01-11T21:53:36.9517104Z Generating XML reports...
2023-01-11T21:53:36.9517547Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213322.xml
2023-01-11T21:53:36.9517914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9518077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9518458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9518644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9518663Z 
2023-01-11T21:53:36.9518767Z Running tests...
2023-01-11T21:53:36.9519025Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9519335Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9519742Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:53:36.9519762Z 
2023-01-11T21:53:36.9520022Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9520118Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9520156Z 
2023-01-11T21:53:36.9520243Z OK (skipped=1)
2023-01-11T21:53:36.9520261Z 
2023-01-11T21:53:36.9520382Z Generating XML reports...
2023-01-11T21:53:36.9520825Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213324.xml
2023-01-11T21:53:36.9521190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9521364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9521740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9521928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9521948Z 
2023-01-11T21:53:36.9522055Z Running tests...
2023-01-11T21:53:36.9522347Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9522761Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9523168Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:53:36.9523188Z 
2023-01-11T21:53:36.9523447Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9523555Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9523574Z 
2023-01-11T21:53:36.9523680Z OK (skipped=1)
2023-01-11T21:53:36.9523699Z 
2023-01-11T21:53:36.9523820Z Generating XML reports...
2023-01-11T21:53:36.9524263Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213327.xml
2023-01-11T21:53:36.9524627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9524782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9525212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9525404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9525424Z 
2023-01-11T21:53:36.9525530Z Running tests...
2023-01-11T21:53:36.9525789Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9526099Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9526404Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:36.9526425Z 
2023-01-11T21:53:36.9526678Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9526788Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9526808Z 
2023-01-11T21:53:36.9526894Z OK (skipped=1)
2023-01-11T21:53:36.9526913Z 
2023-01-11T21:53:36.9527036Z Generating XML reports...
2023-01-11T21:53:36.9527482Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213329.xml
2023-01-11T21:53:36.9527848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9528019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9528394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9528580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9528599Z 
2023-01-11T21:53:36.9528705Z Running tests...
2023-01-11T21:53:36.9528947Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9529259Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9529655Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:53:36.9529678Z 
2023-01-11T21:53:36.9529935Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9530044Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9530063Z 
2023-01-11T21:53:36.9530168Z OK (skipped=1)
2023-01-11T21:53:36.9530187Z 
2023-01-11T21:53:36.9530309Z Generating XML reports...
2023-01-11T21:53:36.9530748Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213332.xml
2023-01-11T21:53:36.9531113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9531268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9531643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9531934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9531954Z 
2023-01-11T21:53:36.9532062Z Running tests...
2023-01-11T21:53:36.9532318Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9532627Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9533033Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:53:36.9533053Z 
2023-01-11T21:53:36.9533309Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9533417Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9533436Z 
2023-01-11T21:53:36.9533522Z OK (skipped=1)
2023-01-11T21:53:36.9533540Z 
2023-01-11T21:53:36.9533662Z Generating XML reports...
2023-01-11T21:53:36.9534102Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213334.xml
2023-01-11T21:53:36.9534475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9534648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9535021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9535207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9535226Z 
2023-01-11T21:53:36.9535331Z Running tests...
2023-01-11T21:53:36.9535588Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9535884Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9536278Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:53:36.9536298Z 
2023-01-11T21:53:36.9536555Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9536670Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9536690Z 
2023-01-11T21:53:36.9536793Z OK (skipped=1)
2023-01-11T21:53:36.9536813Z 
2023-01-11T21:53:36.9536934Z Generating XML reports...
2023-01-11T21:53:36.9537376Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213336.xml
2023-01-11T21:53:36.9537740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9537912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9538275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9538462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9538482Z 
2023-01-11T21:53:36.9538588Z Running tests...
2023-01-11T21:53:36.9538851Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9539162Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9539452Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:36.9539472Z 
2023-01-11T21:53:36.9539729Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9539837Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9539856Z 
2023-01-11T21:53:36.9539943Z OK (skipped=1)
2023-01-11T21:53:36.9539978Z 
2023-01-11T21:53:36.9540083Z Generating XML reports...
2023-01-11T21:53:36.9540525Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213339.xml
2023-01-11T21:53:36.9540894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9541123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9541545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9541739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9541759Z 
2023-01-11T21:53:36.9541866Z Running tests...
2023-01-11T21:53:36.9542126Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9542420Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9542695Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:36.9542714Z 
2023-01-11T21:53:36.9542967Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9543076Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9543096Z 
2023-01-11T21:53:36.9543203Z OK (skipped=1)
2023-01-11T21:53:36.9543225Z 
2023-01-11T21:53:36.9543346Z Generating XML reports...
2023-01-11T21:53:36.9543791Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213341.xml
2023-01-11T21:53:36.9544158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9544332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9544689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9544877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9544896Z 
2023-01-11T21:53:36.9545002Z Running tests...
2023-01-11T21:53:36.9545256Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9545566Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9545843Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:36.9545863Z 
2023-01-11T21:53:36.9546120Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9546231Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9546250Z 
2023-01-11T21:53:36.9546354Z OK (skipped=1)
2023-01-11T21:53:36.9546373Z 
2023-01-11T21:53:36.9546477Z Generating XML reports...
2023-01-11T21:53:36.9546916Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213344.xml
2023-01-11T21:53:36.9547278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9547449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9547821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9548015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9548035Z 
2023-01-11T21:53:36.9548142Z Running tests...
2023-01-11T21:53:36.9548399Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9548691Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9548968Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:36.9548988Z 
2023-01-11T21:53:36.9549239Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9549347Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9549367Z 
2023-01-11T21:53:36.9549471Z OK (skipped=1)
2023-01-11T21:53:36.9549490Z 
2023-01-11T21:53:36.9549610Z Generating XML reports...
2023-01-11T21:53:36.9550052Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213346.xml
2023-01-11T21:53:36.9550528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9550708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9551070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9551260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9551279Z 
2023-01-11T21:53:36.9551383Z Running tests...
2023-01-11T21:53:36.9551642Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9552144Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9552426Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:36.9552446Z 
2023-01-11T21:53:36.9552708Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9552826Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9552845Z 
2023-01-11T21:53:36.9552949Z OK (skipped=1)
2023-01-11T21:53:36.9552968Z 
2023-01-11T21:53:36.9553074Z Generating XML reports...
2023-01-11T21:53:36.9553517Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213348.xml
2023-01-11T21:53:36.9553883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9554055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9554431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9554618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9554637Z 
2023-01-11T21:53:36.9554742Z Running tests...
2023-01-11T21:53:36.9555008Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9555307Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9555578Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:36.9555598Z 
2023-01-11T21:53:36.9555855Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9555965Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9555984Z 
2023-01-11T21:53:36.9556088Z OK (skipped=1)
2023-01-11T21:53:36.9556106Z 
2023-01-11T21:53:36.9556226Z Generating XML reports...
2023-01-11T21:53:36.9556670Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213351.xml
2023-01-11T21:53:36.9557034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9557206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9557575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9557767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9557786Z 
2023-01-11T21:53:36.9557891Z Running tests...
2023-01-11T21:53:36.9558148Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9558459Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9558725Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:36.9558744Z 
2023-01-11T21:53:36.9559001Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9559109Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9559128Z 
2023-01-11T21:53:36.9559233Z OK (skipped=1)
2023-01-11T21:53:36.9559251Z 
2023-01-11T21:53:36.9559440Z Generating XML reports...
2023-01-11T21:53:36.9559940Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213353.xml
2023-01-11T21:53:36.9560316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9560488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9560862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9561050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9561070Z 
2023-01-11T21:53:36.9561177Z Running tests...
2023-01-11T21:53:36.9561433Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9561741Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9562002Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:36.9562021Z 
2023-01-11T21:53:36.9562278Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9562386Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9562405Z 
2023-01-11T21:53:36.9562508Z OK (skipped=1)
2023-01-11T21:53:36.9562527Z 
2023-01-11T21:53:36.9562649Z Generating XML reports...
2023-01-11T21:53:36.9563087Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213356.xml
2023-01-11T21:53:36.9563451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9563623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9563980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9564175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9564195Z 
2023-01-11T21:53:36.9564301Z Running tests...
2023-01-11T21:53:36.9564555Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9564864Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9565128Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:36.9565147Z 
2023-01-11T21:53:36.9565403Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9565513Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9565532Z 
2023-01-11T21:53:36.9565636Z OK (skipped=1)
2023-01-11T21:53:36.9565655Z 
2023-01-11T21:53:36.9565758Z Generating XML reports...
2023-01-11T21:53:36.9566203Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213358.xml
2023-01-11T21:53:36.9566580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9566753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9567130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9567317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9567336Z 
2023-01-11T21:53:36.9567445Z Running tests...
2023-01-11T21:53:36.9567701Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9568009Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9568253Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:36.9568272Z 
2023-01-11T21:53:36.9568528Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9568693Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9568749Z 
2023-01-11T21:53:36.9568858Z OK (skipped=1)
2023-01-11T21:53:36.9568877Z 
2023-01-11T21:53:36.9568998Z Generating XML reports...
2023-01-11T21:53:36.9569442Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213400.xml
2023-01-11T21:53:36.9569808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9569978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9570355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9570528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9570547Z 
2023-01-11T21:53:36.9570653Z Running tests...
2023-01-11T21:53:36.9570911Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9571229Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9571485Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:36.9571504Z 
2023-01-11T21:53:36.9571761Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9571872Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9571890Z 
2023-01-11T21:53:36.9571995Z OK (skipped=1)
2023-01-11T21:53:36.9572014Z 
2023-01-11T21:53:36.9572118Z Generating XML reports...
2023-01-11T21:53:36.9572563Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213403.xml
2023-01-11T21:53:36.9572929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9573103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9573487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9573676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9573694Z 
2023-01-11T21:53:36.9573800Z Running tests...
2023-01-11T21:53:36.9574057Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9574371Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9574631Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2023-01-11T21:53:36.9574667Z 
2023-01-11T21:53:36.9574903Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9575012Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9575031Z 
2023-01-11T21:53:36.9575136Z OK (skipped=1)
2023-01-11T21:53:36.9575155Z 
2023-01-11T21:53:36.9575275Z Generating XML reports...
2023-01-11T21:53:36.9575723Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213405.xml
2023-01-11T21:53:36.9576090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9576261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9576635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9576808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9576826Z 
2023-01-11T21:53:36.9576930Z Running tests...
2023-01-11T21:53:36.9577186Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9577495Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9577830Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2023-01-11T21:53:36.9577893Z 
2023-01-11T21:53:36.9578157Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9578266Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9578285Z 
2023-01-11T21:53:36.9578390Z OK (skipped=1)
2023-01-11T21:53:36.9578409Z 
2023-01-11T21:53:36.9578530Z Generating XML reports...
2023-01-11T21:53:36.9578959Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213407.xml
2023-01-11T21:53:36.9579327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9579497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9579873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9580058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9580084Z 
2023-01-11T21:53:36.9580191Z Running tests...
2023-01-11T21:53:36.9580448Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9580757Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9581006Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:36.9581042Z 
2023-01-11T21:53:36.9581281Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9581388Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9581406Z 
2023-01-11T21:53:36.9581512Z OK (skipped=1)
2023-01-11T21:53:36.9581531Z 
2023-01-11T21:53:36.9581654Z Generating XML reports...
2023-01-11T21:53:36.9582092Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213410.xml
2023-01-11T21:53:36.9582458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9582638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9583012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9583181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9583217Z 
2023-01-11T21:53:36.9583306Z Running tests...
2023-01-11T21:53:36.9583560Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9583872Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9584140Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9584358Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15948
2023-01-11T21:53:36.9584574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15949
2023-01-11T21:53:36.9584943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9585098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9585471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9585658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9586018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9586191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9586565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9586753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9587092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9587334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9587717Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9588113Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9588344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9588572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9588672Z ok (5.594s)
2023-01-11T21:53:36.9588691Z 
2023-01-11T21:53:36.9588951Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9589064Z Ran 1 test in 5.594s
2023-01-11T21:53:36.9589083Z 
2023-01-11T21:53:36.9589175Z OK
2023-01-11T21:53:36.9589195Z 
2023-01-11T21:53:36.9589317Z Generating XML reports...
2023-01-11T21:53:36.9589747Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213412.xml
2023-01-11T21:53:36.9590116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9590290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9590667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9590854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9590873Z 
2023-01-11T21:53:36.9590981Z Running tests...
2023-01-11T21:53:36.9591239Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9591560Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9591804Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:36.9592141Z 
2023-01-11T21:53:36.9592406Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9592517Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9592537Z 
2023-01-11T21:53:36.9592642Z OK (skipped=1)
2023-01-11T21:53:36.9592662Z 
2023-01-11T21:53:36.9592784Z Generating XML reports...
2023-01-11T21:53:36.9593228Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213420.xml
2023-01-11T21:53:36.9593595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9593766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9594146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9594324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9594362Z 
2023-01-11T21:53:36.9594453Z Running tests...
2023-01-11T21:53:36.9594712Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9595024Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9595287Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:36.9595306Z 
2023-01-11T21:53:36.9595563Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9595672Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9595691Z 
2023-01-11T21:53:36.9595797Z OK (skipped=1)
2023-01-11T21:53:36.9595816Z 
2023-01-11T21:53:36.9595935Z Generating XML reports...
2023-01-11T21:53:36.9596440Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213423.xml
2023-01-11T21:53:36.9596867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9597037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9597411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9597598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9597618Z 
2023-01-11T21:53:36.9597723Z Running tests...
2023-01-11T21:53:36.9597981Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9598291Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9598544Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:36.9598583Z 
2023-01-11T21:53:36.9598828Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9598939Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9598958Z 
2023-01-11T21:53:36.9599062Z OK (skipped=1)
2023-01-11T21:53:36.9599080Z 
2023-01-11T21:53:36.9599200Z Generating XML reports...
2023-01-11T21:53:36.9599644Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213425.xml
2023-01-11T21:53:36.9600005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9600175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9600544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9600737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9600760Z 
2023-01-11T21:53:36.9600868Z Running tests...
2023-01-11T21:53:36.9601109Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9601420Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9601682Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9601902Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16162
2023-01-11T21:53:36.9602116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16163
2023-01-11T21:53:36.9602481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9602650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9603023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9603218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9603567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9603739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9604105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9604291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9604538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9604776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9605174Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9605619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9605875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9606105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9606442Z STAGE:2023-01-11 21:34:32 16162:16162 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9606778Z STAGE:2023-01-11 21:34:33 16163:16163 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9607114Z STAGE:2023-01-11 21:34:33 16163:16163 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9607448Z STAGE:2023-01-11 21:34:33 16162:16162 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9607798Z STAGE:2023-01-11 21:34:33 16163:16163 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9608146Z STAGE:2023-01-11 21:34:33 16162:16162 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9608473Z STAGE:2023-01-11 21:34:33 16163:16163 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9608780Z STAGE:2023-01-11 21:34:33 16162:16162 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9609115Z STAGE:2023-01-11 21:34:33 16163:16163 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9609461Z STAGE:2023-01-11 21:34:33 16163:16163 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9609787Z STAGE:2023-01-11 21:34:33 16162:16162 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9610126Z STAGE:2023-01-11 21:34:33 16162:16162 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9610451Z STAGE:2023-01-11 21:34:33 16163:16163 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9610785Z STAGE:2023-01-11 21:34:33 16162:16162 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9611119Z STAGE:2023-01-11 21:34:33 16163:16163 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9611459Z STAGE:2023-01-11 21:34:33 16163:16163 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9611766Z STAGE:2023-01-11 21:34:33 16162:16162 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9612111Z STAGE:2023-01-11 21:34:33 16162:16162 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9612213Z ok (5.705s)
2023-01-11T21:53:36.9612232Z 
2023-01-11T21:53:36.9612492Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9612602Z Ran 1 test in 5.706s
2023-01-11T21:53:36.9612621Z 
2023-01-11T21:53:36.9612712Z OK
2023-01-11T21:53:36.9612731Z 
2023-01-11T21:53:36.9612854Z Generating XML reports...
2023-01-11T21:53:36.9613311Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213427.xml
2023-01-11T21:53:36.9613678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9613835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9614216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9614407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9614426Z 
2023-01-11T21:53:36.9614533Z Running tests...
2023-01-11T21:53:36.9614793Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9615103Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9615418Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9615694Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16280
2023-01-11T21:53:36.9615889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16281
2023-01-11T21:53:36.9616259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9616431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9616808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9616995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9617358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9617530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9617915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9618106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9618335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9618577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9618977Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9619374Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9619602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9619829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9620173Z STAGE:2023-01-11 21:34:41 16281:16281 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9620505Z STAGE:2023-01-11 21:34:41 16280:16280 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9620842Z STAGE:2023-01-11 21:34:41 16281:16281 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9621159Z STAGE:2023-01-11 21:34:41 16280:16280 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9621505Z STAGE:2023-01-11 21:34:41 16281:16281 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9621853Z STAGE:2023-01-11 21:34:41 16280:16280 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9622177Z STAGE:2023-01-11 21:34:41 16281:16281 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9622553Z STAGE:2023-01-11 21:34:41 16280:16280 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9622894Z STAGE:2023-01-11 21:34:41 16281:16281 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9623240Z STAGE:2023-01-11 21:34:41 16281:16281 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9623568Z STAGE:2023-01-11 21:34:41 16280:16280 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9623907Z STAGE:2023-01-11 21:34:41 16280:16280 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9624217Z STAGE:2023-01-11 21:34:41 16281:16281 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9624541Z STAGE:2023-01-11 21:34:41 16280:16280 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9624873Z STAGE:2023-01-11 21:34:41 16281:16281 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9625263Z STAGE:2023-01-11 21:34:41 16281:16281 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9625638Z STAGE:2023-01-11 21:34:41 16280:16280 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9625977Z STAGE:2023-01-11 21:34:41 16280:16280 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9626081Z ok (5.660s)
2023-01-11T21:53:36.9626101Z 
2023-01-11T21:53:36.9626362Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9626455Z Ran 1 test in 5.660s
2023-01-11T21:53:36.9626493Z 
2023-01-11T21:53:36.9626566Z OK
2023-01-11T21:53:36.9626585Z 
2023-01-11T21:53:36.9626705Z Generating XML reports...
2023-01-11T21:53:36.9627157Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213436.xml
2023-01-11T21:53:36.9627525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9627706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9628086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9628272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9628292Z 
2023-01-11T21:53:36.9628401Z Running tests...
2023-01-11T21:53:36.9628643Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9628958Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9629234Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9629454Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16398
2023-01-11T21:53:36.9629664Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16399
2023-01-11T21:53:36.9630038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9630212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9630591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9630762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9631130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9631302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9631677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9632095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9632357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9632604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9633015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9633413Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9633625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9633853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9634192Z STAGE:2023-01-11 21:34:49 16399:16399 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9634526Z STAGE:2023-01-11 21:34:49 16398:16398 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9634935Z STAGE:2023-01-11 21:34:49 16399:16399 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9635340Z STAGE:2023-01-11 21:34:49 16398:16398 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9635686Z STAGE:2023-01-11 21:34:49 16399:16399 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9636036Z STAGE:2023-01-11 21:34:49 16398:16398 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9636364Z STAGE:2023-01-11 21:34:49 16399:16399 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9636668Z STAGE:2023-01-11 21:34:49 16398:16398 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9637000Z STAGE:2023-01-11 21:34:49 16399:16399 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9637344Z STAGE:2023-01-11 21:34:49 16399:16399 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9637679Z STAGE:2023-01-11 21:34:49 16398:16398 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9638019Z STAGE:2023-01-11 21:34:49 16398:16398 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9638345Z STAGE:2023-01-11 21:34:49 16399:16399 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9638672Z STAGE:2023-01-11 21:34:49 16398:16398 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9639005Z STAGE:2023-01-11 21:34:49 16399:16399 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9639350Z STAGE:2023-01-11 21:34:49 16399:16399 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9639664Z STAGE:2023-01-11 21:34:49 16398:16398 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9640006Z STAGE:2023-01-11 21:34:49 16398:16398 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9640112Z ok (5.697s)
2023-01-11T21:53:36.9640136Z 
2023-01-11T21:53:36.9640402Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9640514Z Ran 1 test in 5.698s
2023-01-11T21:53:36.9640533Z 
2023-01-11T21:53:36.9640625Z OK
2023-01-11T21:53:36.9640644Z 
2023-01-11T21:53:36.9640767Z Generating XML reports...
2023-01-11T21:53:36.9641218Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213444.xml
2023-01-11T21:53:36.9641589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9641747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9642124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9642315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9642337Z 
2023-01-11T21:53:36.9642447Z Running tests...
2023-01-11T21:53:36.9642708Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9643022Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9643266Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2023-01-11T21:53:36.9643286Z 
2023-01-11T21:53:36.9643543Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9643635Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9643673Z 
2023-01-11T21:53:36.9643762Z OK (skipped=1)
2023-01-11T21:53:36.9643781Z 
2023-01-11T21:53:36.9643903Z Generating XML reports...
2023-01-11T21:53:36.9644352Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213452.xml
2023-01-11T21:53:36.9644719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9644990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9645376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9645566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9645586Z 
2023-01-11T21:53:36.9645693Z Running tests...
2023-01-11T21:53:36.9645936Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9646248Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9646502Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2023-01-11T21:53:36.9646521Z 
2023-01-11T21:53:36.9646774Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9646883Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9646906Z 
2023-01-11T21:53:36.9647011Z OK (skipped=1)
2023-01-11T21:53:36.9647032Z 
2023-01-11T21:53:36.9647151Z Generating XML reports...
2023-01-11T21:53:36.9647596Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213455.xml
2023-01-11T21:53:36.9647969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9648126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9648501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9648686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9648706Z 
2023-01-11T21:53:36.9648808Z Running tests...
2023-01-11T21:53:36.9649063Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9649372Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9649634Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9649849Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16582
2023-01-11T21:53:36.9650043Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16583
2023-01-11T21:53:36.9650409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9650580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9650953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9651139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9651499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9651680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9652048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9652231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9652462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9652698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9653097Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9653490Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9653720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9654072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9654179Z ok (5.701s)
2023-01-11T21:53:36.9654199Z 
2023-01-11T21:53:36.9654464Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9654556Z Ran 1 test in 5.701s
2023-01-11T21:53:36.9654591Z 
2023-01-11T21:53:36.9654666Z OK
2023-01-11T21:53:36.9654684Z 
2023-01-11T21:53:36.9654806Z Generating XML reports...
2023-01-11T21:53:36.9655253Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213457.xml
2023-01-11T21:53:36.9655621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9655794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9656169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9656364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9656384Z 
2023-01-11T21:53:36.9656488Z Running tests...
2023-01-11T21:53:36.9656732Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9657046Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9657312Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9657531Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16699
2023-01-11T21:53:36.9657739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16700
2023-01-11T21:53:36.9658103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9658272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9658652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9658825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9659189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9659358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9659728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9659914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9660156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9660398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9660802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9661197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9661410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9661635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9662522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_comparison.py:992: UserWarning: Specified kernel cache directory could not be created! This disables kernel caching. Specified directory is /var/lib/jenkins/.cache/torch/kernels. This warning will appear only once per process. (Triggered internally at /var/lib/jenkins/workspace/aten/src/ATen/native/cuda/jit_utils.cpp:1444.)
2023-01-11T21:53:36.9662647Z   matches = torch.isclose(
2023-01-11T21:53:36.9662746Z ok (5.978s)
2023-01-11T21:53:36.9662815Z 
2023-01-11T21:53:36.9663121Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9663237Z Ran 1 test in 5.978s
2023-01-11T21:53:36.9663257Z 
2023-01-11T21:53:36.9663348Z OK
2023-01-11T21:53:36.9663367Z 
2023-01-11T21:53:36.9663490Z Generating XML reports...
2023-01-11T21:53:36.9663926Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213505.xml
2023-01-11T21:53:36.9664297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9664471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9664847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9665037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9665056Z 
2023-01-11T21:53:36.9665169Z Running tests...
2023-01-11T21:53:36.9665433Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9665747Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9666003Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2023-01-11T21:53:36.9666024Z 
2023-01-11T21:53:36.9666261Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9666372Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9666391Z 
2023-01-11T21:53:36.9666497Z OK (skipped=1)
2023-01-11T21:53:36.9666517Z 
2023-01-11T21:53:36.9666639Z Generating XML reports...
2023-01-11T21:53:36.9667089Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213514.xml
2023-01-11T21:53:36.9667455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9667635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9668012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9668199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9668219Z 
2023-01-11T21:53:36.9668308Z Running tests...
2023-01-11T21:53:36.9668563Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9668873Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9669141Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9669361Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16849
2023-01-11T21:53:36.9669573Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16850
2023-01-11T21:53:36.9669947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9670120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9670475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9670665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9671027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9671199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9671575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9671760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9672240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9672625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9673050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9673428Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9673661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9673902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:53:36.9674130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9674369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:53:36.9674770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:36.9675165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:36.9675263Z ok (5.821s)
2023-01-11T21:53:36.9675284Z 
2023-01-11T21:53:36.9700555Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9700710Z Ran 1 test in 5.822s
2023-01-11T21:53:36.9700733Z 
2023-01-11T21:53:36.9700814Z OK
2023-01-11T21:53:36.9700833Z 
2023-01-11T21:53:36.9700953Z Generating XML reports...
2023-01-11T21:53:36.9701457Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213516.xml
2023-01-11T21:53:36.9701843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9702013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9702416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9702604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9702624Z 
2023-01-11T21:53:36.9702723Z Running tests...
2023-01-11T21:53:36.9702990Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9703302Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9703556Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2023-01-11T21:53:36.9703577Z 
2023-01-11T21:53:36.9703839Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9703943Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9703963Z 
2023-01-11T21:53:36.9704060Z OK (skipped=1)
2023-01-11T21:53:36.9704079Z 
2023-01-11T21:53:36.9704193Z Generating XML reports...
2023-01-11T21:53:36.9704660Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213525.xml
2023-01-11T21:53:36.9705038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9705207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9705589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9705776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9705795Z 
2023-01-11T21:53:36.9705891Z Running tests...
2023-01-11T21:53:36.9706155Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9706474Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9706733Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9707138Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17001
2023-01-11T21:53:36.9707361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17002
2023-01-11T21:53:36.9707748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9707911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9708308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9708496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9708882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9709047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9709451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9709638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9709888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9710125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9710543Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9710956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9711183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9711416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9711588Z skip: Skipped due to small world size. (4.294s)
2023-01-11T21:53:36.9711609Z 
2023-01-11T21:53:36.9712115Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9712244Z Ran 1 test in 4.294s
2023-01-11T21:53:36.9712265Z 
2023-01-11T21:53:36.9712362Z OK (skipped=1)
2023-01-11T21:53:36.9712381Z 
2023-01-11T21:53:36.9712489Z Generating XML reports...
2023-01-11T21:53:36.9712943Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213527.xml
2023-01-11T21:53:36.9713300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9713462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9713826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9714002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9714030Z 
2023-01-11T21:53:36.9714126Z Running tests...
2023-01-11T21:53:36.9714381Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9714678Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9714944Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T21:53:36.9714965Z 
2023-01-11T21:53:36.9715211Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9715308Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9715327Z 
2023-01-11T21:53:36.9715420Z OK (skipped=1)
2023-01-11T21:53:36.9715439Z 
2023-01-11T21:53:36.9715546Z Generating XML reports...
2023-01-11T21:53:36.9715974Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213534.xml
2023-01-11T21:53:36.9716497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9716666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9717120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9717292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9717311Z 
2023-01-11T21:53:36.9717407Z Running tests...
2023-01-11T21:53:36.9717662Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9717959Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9718248Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T21:53:36.9718268Z 
2023-01-11T21:53:36.9718517Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9718628Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9718647Z 
2023-01-11T21:53:36.9718742Z OK (skipped=1)
2023-01-11T21:53:36.9718761Z 
2023-01-11T21:53:36.9718864Z Generating XML reports...
2023-01-11T21:53:36.9719307Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213536.xml
2023-01-11T21:53:36.9719660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9719827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9720205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9720382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9720401Z 
2023-01-11T21:53:36.9720495Z Running tests...
2023-01-11T21:53:36.9720752Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9721056Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9721315Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9722067Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81030 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.661s)
2023-01-11T21:53:36.9722097Z 
2023-01-11T21:53:36.9722381Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9722488Z Ran 1 test in 1.661s
2023-01-11T21:53:36.9722508Z 
2023-01-11T21:53:36.9722612Z OK (skipped=1)
2023-01-11T21:53:36.9722631Z 
2023-01-11T21:53:36.9722742Z Generating XML reports...
2023-01-11T21:53:36.9723191Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213539.xml
2023-01-11T21:53:36.9723546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9723714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9724079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9724251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9724270Z 
2023-01-11T21:53:36.9724370Z Running tests...
2023-01-11T21:53:36.9724618Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9724921Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9725200Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9725506Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17206
2023-01-11T21:53:36.9725720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17207
2023-01-11T21:53:36.9726073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9726229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9726600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9726776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9727124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9727284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9727650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9727830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9728069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9728300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9728676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9729057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9729276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9729502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9729830Z STAGE:2023-01-11 21:35:48 17207:17207 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9730147Z STAGE:2023-01-11 21:35:48 17206:17206 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:36.9730465Z STAGE:2023-01-11 21:35:49 17206:17206 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9730791Z STAGE:2023-01-11 21:35:49 17207:17207 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:36.9731124Z STAGE:2023-01-11 21:35:49 17206:17206 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9731451Z STAGE:2023-01-11 21:35:49 17207:17207 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:36.9731548Z ok (6.541s)
2023-01-11T21:53:36.9731568Z 
2023-01-11T21:53:36.9731818Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9731923Z Ran 1 test in 6.541s
2023-01-11T21:53:36.9731945Z 
2023-01-11T21:53:36.9732023Z OK
2023-01-11T21:53:36.9732042Z 
2023-01-11T21:53:36.9732154Z Generating XML reports...
2023-01-11T21:53:36.9732598Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213543.xml
2023-01-11T21:53:36.9732962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9733119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9733482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9733666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9733685Z 
2023-01-11T21:53:36.9733782Z Running tests...
2023-01-11T21:53:36.9734037Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9734337Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9735025Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T21:53:36.9735048Z 
2023-01-11T21:53:36.9735335Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9735437Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9735456Z 
2023-01-11T21:53:36.9735544Z OK (skipped=1)
2023-01-11T21:53:36.9735575Z 
2023-01-11T21:53:36.9735678Z Generating XML reports...
2023-01-11T21:53:36.9736116Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213552.xml
2023-01-11T21:53:36.9736473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9736641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9737008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9737197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9737216Z 
2023-01-11T21:53:36.9737322Z Running tests...
2023-01-11T21:53:36.9737569Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9737861Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9738146Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9738882Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81004 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.646s)
2023-01-11T21:53:36.9738903Z 
2023-01-11T21:53:36.9739159Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9739262Z Ran 1 test in 1.646s
2023-01-11T21:53:36.9739282Z 
2023-01-11T21:53:36.9739383Z OK (skipped=1)
2023-01-11T21:53:36.9739402Z 
2023-01-11T21:53:36.9739513Z Generating XML reports...
2023-01-11T21:53:36.9739951Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213554.xml
2023-01-11T21:53:36.9740305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9740474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9740834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9741014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9741034Z 
2023-01-11T21:53:36.9741134Z Running tests...
2023-01-11T21:53:36.9741389Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9741695Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9741978Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T21:53:36.9741997Z 
2023-01-11T21:53:36.9742256Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9742364Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9742383Z 
2023-01-11T21:53:36.9742470Z OK (skipped=1)
2023-01-11T21:53:36.9742506Z 
2023-01-11T21:53:36.9742609Z Generating XML reports...
2023-01-11T21:53:36.9743048Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213558.xml
2023-01-11T21:53:36.9743412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9743681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9744065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9744254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9744274Z 
2023-01-11T21:53:36.9744381Z Running tests...
2023-01-11T21:53:36.9744636Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9744932Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9745216Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9745435Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17427
2023-01-11T21:53:36.9745649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17428
2023-01-11T21:53:36.9746022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9746194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9746569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9746756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9747114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9747268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9747633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9747819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9748063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9748310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9748704Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9749092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9749322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9749528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9749686Z skip: Skipped due to small world size. (4.379s)
2023-01-11T21:53:36.9749705Z 
2023-01-11T21:53:36.9749964Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9750074Z Ran 1 test in 4.380s
2023-01-11T21:53:36.9750093Z 
2023-01-11T21:53:36.9750202Z OK (skipped=1)
2023-01-11T21:53:36.9750221Z 
2023-01-11T21:53:36.9750346Z Generating XML reports...
2023-01-11T21:53:36.9750791Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213601.xml
2023-01-11T21:53:36.9751156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9751327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9751682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9752070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9752093Z 
2023-01-11T21:53:36.9752205Z Running tests...
2023-01-11T21:53:36.9752473Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9752780Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9753205Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T21:53:36.9753227Z 
2023-01-11T21:53:36.9753494Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9753603Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9753622Z 
2023-01-11T21:53:36.9753727Z OK (skipped=1)
2023-01-11T21:53:36.9753746Z 
2023-01-11T21:53:36.9753849Z Generating XML reports...
2023-01-11T21:53:36.9754291Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213608.xml
2023-01-11T21:53:36.9754655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9754828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9755201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9755396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9755416Z 
2023-01-11T21:53:36.9755520Z Running tests...
2023-01-11T21:53:36.9755776Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9756065Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9756359Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T21:53:36.9756378Z 
2023-01-11T21:53:36.9756634Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9756743Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9756762Z 
2023-01-11T21:53:36.9756865Z OK (skipped=1)
2023-01-11T21:53:36.9756884Z 
2023-01-11T21:53:36.9757004Z Generating XML reports...
2023-01-11T21:53:36.9757448Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213610.xml
2023-01-11T21:53:36.9757814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9757987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9758346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9758535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9758554Z 
2023-01-11T21:53:36.9758659Z Running tests...
2023-01-11T21:53:36.9758913Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9759220Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9759498Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9759722Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17598
2023-01-11T21:53:36.9759936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17599
2023-01-11T21:53:36.9760299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9760455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9760826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9761012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9761368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9761540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9761953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9762183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9762426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9762652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9763048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9763440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9763667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9763891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9763991Z ok (5.785s)
2023-01-11T21:53:36.9764015Z 
2023-01-11T21:53:36.9764279Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9764391Z Ran 1 test in 5.785s
2023-01-11T21:53:36.9764411Z 
2023-01-11T21:53:36.9764502Z OK
2023-01-11T21:53:36.9764522Z 
2023-01-11T21:53:36.9764625Z Generating XML reports...
2023-01-11T21:53:36.9765066Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213613.xml
2023-01-11T21:53:36.9765429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9765602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9765974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9766162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9766182Z 
2023-01-11T21:53:36.9766293Z Running tests...
2023-01-11T21:53:36.9766555Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9766863Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9767138Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9767355Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17715
2023-01-11T21:53:36.9767568Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17716
2023-01-11T21:53:36.9767931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9768103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9768478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9768673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9769031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9769183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9769550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9769733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9769977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9770220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9770612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9771046Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9771329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9771554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9771638Z ok (5.762s)
2023-01-11T21:53:36.9771658Z 
2023-01-11T21:53:36.9771919Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9772029Z Ran 1 test in 5.762s
2023-01-11T21:53:36.9772048Z 
2023-01-11T21:53:36.9772139Z OK
2023-01-11T21:53:36.9772158Z 
2023-01-11T21:53:36.9772279Z Generating XML reports...
2023-01-11T21:53:36.9772725Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213621.xml
2023-01-11T21:53:36.9773086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9773266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9773640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9773812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9773832Z 
2023-01-11T21:53:36.9773938Z Running tests...
2023-01-11T21:53:36.9774196Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9774501Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9774798Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T21:53:36.9774817Z 
2023-01-11T21:53:36.9775072Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9775181Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9775199Z 
2023-01-11T21:53:36.9775310Z OK (skipped=1)
2023-01-11T21:53:36.9775329Z 
2023-01-11T21:53:36.9775434Z Generating XML reports...
2023-01-11T21:53:36.9775876Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213629.xml
2023-01-11T21:53:36.9776238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9776409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9776781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9776969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9776988Z 
2023-01-11T21:53:36.9777093Z Running tests...
2023-01-11T21:53:36.9777351Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9777657Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9777939Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9778157Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17865
2023-01-11T21:53:36.9778371Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17866
2023-01-11T21:53:36.9778735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9778905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9779275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9779465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9779823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9780085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9780442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9780628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9780873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9781115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9781510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9781902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9782133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9782381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:53:36.9782587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9782825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:53:36.9783217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:36.9783609Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:36.9783710Z ok (5.753s)
2023-01-11T21:53:36.9783729Z 
2023-01-11T21:53:36.9783989Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9784097Z Ran 1 test in 5.754s
2023-01-11T21:53:36.9784117Z 
2023-01-11T21:53:36.9784208Z OK
2023-01-11T21:53:36.9784231Z 
2023-01-11T21:53:36.9784354Z Generating XML reports...
2023-01-11T21:53:36.9784784Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213632.xml
2023-01-11T21:53:36.9785151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9785325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9785701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9785890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9785909Z 
2023-01-11T21:53:36.9786015Z Running tests...
2023-01-11T21:53:36.9786273Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9786583Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9786881Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T21:53:36.9786902Z 
2023-01-11T21:53:36.9787143Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9787252Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9787272Z 
2023-01-11T21:53:36.9787376Z OK (skipped=1)
2023-01-11T21:53:36.9787395Z 
2023-01-11T21:53:36.9787515Z Generating XML reports...
2023-01-11T21:53:36.9787961Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213640.xml
2023-01-11T21:53:36.9788325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9788501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9788875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9789161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9789182Z 
2023-01-11T21:53:36.9789275Z Running tests...
2023-01-11T21:53:36.9789540Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9789852Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9790142Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9790361Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18017
2023-01-11T21:53:36.9790577Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18018
2023-01-11T21:53:36.9790942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9791113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9791476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9791664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9792304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9792486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9792871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9793057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9793304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9793548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9793952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9794337Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9794567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9794794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9794949Z skip: Skipped due to small world size. (4.250s)
2023-01-11T21:53:36.9794969Z 
2023-01-11T21:53:36.9795233Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9795344Z Ran 1 test in 4.250s
2023-01-11T21:53:36.9795363Z 
2023-01-11T21:53:36.9795467Z OK (skipped=1)
2023-01-11T21:53:36.9795486Z 
2023-01-11T21:53:36.9795606Z Generating XML reports...
2023-01-11T21:53:36.9796036Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213642.xml
2023-01-11T21:53:36.9796414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9796587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9796962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9797150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9797170Z 
2023-01-11T21:53:36.9797277Z Running tests...
2023-01-11T21:53:36.9797536Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9797846Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9798112Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9798312Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18122
2023-01-11T21:53:36.9798705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18123
2023-01-11T21:53:36.9799086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9799258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9799630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9799818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9800177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9800347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9800716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9800894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9801140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9801385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9801781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9802179Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9802411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9802637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9802876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:53:36.9803108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:53:36.9803502Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:36.9803890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:36.9803989Z ok (5.693s)
2023-01-11T21:53:36.9804009Z 
2023-01-11T21:53:36.9804269Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9804378Z Ran 1 test in 5.693s
2023-01-11T21:53:36.9804397Z 
2023-01-11T21:53:36.9804487Z OK
2023-01-11T21:53:36.9804506Z 
2023-01-11T21:53:36.9804627Z Generating XML reports...
2023-01-11T21:53:36.9805080Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213649.xml
2023-01-11T21:53:36.9805440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9805616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9805993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9806184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9806203Z 
2023-01-11T21:53:36.9806308Z Running tests...
2023-01-11T21:53:36.9806565Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9806878Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9807140Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9807355Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18246
2023-01-11T21:53:36.9807655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18247
2023-01-11T21:53:36.9808030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9808203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9808580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9808770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9809129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9809303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9809676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9809844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9810096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9810339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9810736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9811131Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9811361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9811590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9811736Z skip: Need at least 3 CUDA devices (4.309s)
2023-01-11T21:53:36.9811756Z 
2023-01-11T21:53:36.9812017Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9812115Z Ran 1 test in 4.309s
2023-01-11T21:53:36.9812137Z 
2023-01-11T21:53:36.9812246Z OK (skipped=1)
2023-01-11T21:53:36.9812265Z 
2023-01-11T21:53:36.9812387Z Generating XML reports...
2023-01-11T21:53:36.9812834Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213657.xml
2023-01-11T21:53:36.9813201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9813373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9813748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9813935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9813954Z 
2023-01-11T21:53:36.9814061Z Running tests...
2023-01-11T21:53:36.9814301Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9814619Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9814869Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2023-01-11T21:53:36.9814889Z 
2023-01-11T21:53:36.9815149Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9815258Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9815276Z 
2023-01-11T21:53:36.9815380Z OK (skipped=1)
2023-01-11T21:53:36.9815398Z 
2023-01-11T21:53:36.9815519Z Generating XML reports...
2023-01-11T21:53:36.9815964Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213704.xml
2023-01-11T21:53:36.9816312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9816487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9816959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9817154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9817174Z 
2023-01-11T21:53:36.9817279Z Running tests...
2023-01-11T21:53:36.9817541Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9817852Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9818102Z   test_barrier (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2023-01-11T21:53:36.9818122Z 
2023-01-11T21:53:36.9818378Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9818471Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9818490Z 
2023-01-11T21:53:36.9818594Z OK (skipped=1)
2023-01-11T21:53:36.9818613Z 
2023-01-11T21:53:36.9818735Z Generating XML reports...
2023-01-11T21:53:36.9819189Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213706.xml
2023-01-11T21:53:36.9819558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9819730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9820105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9820291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9820310Z 
2023-01-11T21:53:36.9820416Z Running tests...
2023-01-11T21:53:36.9820654Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9820965Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9821214Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9821439Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18417
2023-01-11T21:53:36.9821655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18418
2023-01-11T21:53:36.9822023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9822198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9822626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9822797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9823158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9823327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9823699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9823888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9824134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9824380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9824779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9825220Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9825434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9825662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9825821Z ok (6.461s)
2023-01-11T21:53:36.9825840Z 
2023-01-11T21:53:36.9826142Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9826258Z Ran 1 test in 6.461s
2023-01-11T21:53:36.9826277Z 
2023-01-11T21:53:36.9826368Z OK
2023-01-11T21:53:36.9826387Z 
2023-01-11T21:53:36.9826510Z Generating XML reports...
2023-01-11T21:53:36.9826960Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213709.xml
2023-01-11T21:53:36.9827310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9827486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9827861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9828050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9828069Z 
2023-01-11T21:53:36.9828176Z Running tests...
2023-01-11T21:53:36.9828435Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9828747Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9829014Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2023-01-11T21:53:36.9829034Z 
2023-01-11T21:53:36.9829291Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9829384Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9829403Z 
2023-01-11T21:53:36.9829510Z OK (skipped=1)
2023-01-11T21:53:36.9829528Z 
2023-01-11T21:53:36.9829647Z Generating XML reports...
2023-01-11T21:53:36.9830086Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213718.xml
2023-01-11T21:53:36.9830448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9830632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9831006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9831193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9831212Z 
2023-01-11T21:53:36.9831317Z Running tests...
2023-01-11T21:53:36.9831555Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9832048Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9832334Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9832556Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18565
2023-01-11T21:53:36.9832773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18566
2023-01-11T21:53:36.9833153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9833327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9833706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9833876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9834236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9834405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9834776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9834963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9835210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9835598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9836007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9836405Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9836618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9836844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9837000Z skip: Skipped due to small world size. (4.235s)
2023-01-11T21:53:36.9837020Z 
2023-01-11T21:53:36.9837282Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9837392Z Ran 1 test in 4.235s
2023-01-11T21:53:36.9837415Z 
2023-01-11T21:53:36.9837521Z OK (skipped=1)
2023-01-11T21:53:36.9837542Z 
2023-01-11T21:53:36.9837664Z Generating XML reports...
2023-01-11T21:53:36.9838111Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213720.xml
2023-01-11T21:53:36.9838481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9838638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9839013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9839200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9839219Z 
2023-01-11T21:53:36.9839325Z Running tests...
2023-01-11T21:53:36.9839581Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9839890Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9840154Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2023-01-11T21:53:36.9840173Z 
2023-01-11T21:53:36.9840432Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9840524Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9840560Z 
2023-01-11T21:53:36.9840647Z OK (skipped=1)
2023-01-11T21:53:36.9840666Z 
2023-01-11T21:53:36.9840786Z Generating XML reports...
2023-01-11T21:53:36.9841229Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213727.xml
2023-01-11T21:53:36.9841597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9841769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9842143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9842337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9842356Z 
2023-01-11T21:53:36.9842462Z Running tests...
2023-01-11T21:53:36.9842701Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9843011Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9843272Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9843492Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18703
2023-01-11T21:53:36.9843709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18704
2023-01-11T21:53:36.9844075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9844245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9844725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9844902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9845265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9845437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9845805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9845986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9846231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9846475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9846878Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9847277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9847489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9847714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9847869Z skip: Skipped due to small world size. (4.290s)
2023-01-11T21:53:36.9847889Z 
2023-01-11T21:53:36.9848149Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9848258Z Ran 1 test in 4.291s
2023-01-11T21:53:36.9848277Z 
2023-01-11T21:53:36.9848384Z OK (skipped=1)
2023-01-11T21:53:36.9848402Z 
2023-01-11T21:53:36.9848526Z Generating XML reports...
2023-01-11T21:53:36.9848974Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213729.xml
2023-01-11T21:53:36.9849346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9849502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9849878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9850066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9850085Z 
2023-01-11T21:53:36.9850190Z Running tests...
2023-01-11T21:53:36.9850449Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9850759Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9851035Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2023-01-11T21:53:36.9851058Z 
2023-01-11T21:53:36.9851315Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9851408Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9851444Z 
2023-01-11T21:53:36.9851531Z OK (skipped=1)
2023-01-11T21:53:36.9851550Z 
2023-01-11T21:53:36.9851672Z Generating XML reports...
2023-01-11T21:53:36.9852117Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213736.xml
2023-01-11T21:53:36.9852485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9852659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9853029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9853213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9853232Z 
2023-01-11T21:53:36.9853395Z Running tests...
2023-01-11T21:53:36.9853678Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9853997Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9854271Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2023-01-11T21:53:36.9854291Z 
2023-01-11T21:53:36.9854547Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9854657Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9854677Z 
2023-01-11T21:53:36.9854783Z OK (skipped=1)
2023-01-11T21:53:36.9854802Z 
2023-01-11T21:53:36.9854922Z Generating XML reports...
2023-01-11T21:53:36.9855364Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213738.xml
2023-01-11T21:53:36.9855727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9855888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9856261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9856450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9856469Z 
2023-01-11T21:53:36.9856574Z Running tests...
2023-01-11T21:53:36.9856831Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9857143Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9857415Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2023-01-11T21:53:36.9857434Z 
2023-01-11T21:53:36.9857689Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9857798Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9857817Z 
2023-01-11T21:53:36.9857909Z OK (skipped=1)
2023-01-11T21:53:36.9857927Z 
2023-01-11T21:53:36.9858051Z Generating XML reports...
2023-01-11T21:53:36.9858497Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213741.xml
2023-01-11T21:53:36.9858864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9859035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9859409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9859598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9859618Z 
2023-01-11T21:53:36.9859724Z Running tests...
2023-01-11T21:53:36.9859964Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9860274Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9860536Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2023-01-11T21:53:36.9860556Z 
2023-01-11T21:53:36.9860812Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9860923Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9860941Z 
2023-01-11T21:53:36.9861048Z OK (skipped=1)
2023-01-11T21:53:36.9861066Z 
2023-01-11T21:53:36.9861185Z Generating XML reports...
2023-01-11T21:53:36.9861628Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213743.xml
2023-01-11T21:53:36.9861994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9862150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9862525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9862809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9862830Z 
2023-01-11T21:53:36.9862939Z Running tests...
2023-01-11T21:53:36.9863199Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9863511Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9863776Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2023-01-11T21:53:36.9863795Z 
2023-01-11T21:53:36.9864050Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9864158Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9864177Z 
2023-01-11T21:53:36.9864264Z OK (skipped=1)
2023-01-11T21:53:36.9864283Z 
2023-01-11T21:53:36.9864404Z Generating XML reports...
2023-01-11T21:53:36.9864850Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213746.xml
2023-01-11T21:53:36.9865223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9865396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9865771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9865957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9865976Z 
2023-01-11T21:53:36.9866083Z Running tests...
2023-01-11T21:53:36.9866324Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9866634Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9866921Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9867140Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18973
2023-01-11T21:53:36.9867365Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18974
2023-01-11T21:53:36.9867730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9867907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9868281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9868468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9868812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9868984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9869353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9869545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9869790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9870035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9870431Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9870825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9871055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9871266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9871505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:53:36.9872077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:53:36.9872518Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:36.9872912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:36.9873155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:53:36.9873397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:53:36.9873791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:53:36.9874185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:53:36.9874274Z ok (4.474s)
2023-01-11T21:53:36.9874297Z 
2023-01-11T21:53:36.9874568Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9874677Z Ran 1 test in 4.474s
2023-01-11T21:53:36.9874696Z 
2023-01-11T21:53:36.9874785Z OK
2023-01-11T21:53:36.9874804Z 
2023-01-11T21:53:36.9874924Z Generating XML reports...
2023-01-11T21:53:36.9875370Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213748.xml
2023-01-11T21:53:36.9875737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9875911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9876286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9876459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9876481Z 
2023-01-11T21:53:36.9876593Z Running tests...
2023-01-11T21:53:36.9876852Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9877165Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9877431Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9877651Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19086
2023-01-11T21:53:36.9877866Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19087
2023-01-11T21:53:36.9878230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9878387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9878765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9878957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9879318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9879491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9879861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9880046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9880294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9880542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9880922Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9881454Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9881691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9881921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9882022Z ok (5.249s)
2023-01-11T21:53:36.9882042Z 
2023-01-11T21:53:36.9882304Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9882415Z Ran 1 test in 5.250s
2023-01-11T21:53:36.9882434Z 
2023-01-11T21:53:36.9882522Z OK
2023-01-11T21:53:36.9882542Z 
2023-01-11T21:53:36.9882646Z Generating XML reports...
2023-01-11T21:53:36.9883092Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213755.xml
2023-01-11T21:53:36.9883459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9883640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9884013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9884201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9884219Z 
2023-01-11T21:53:36.9884325Z Running tests...
2023-01-11T21:53:36.9884581Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9884893Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9885158Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9885377Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19202
2023-01-11T21:53:36.9885591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19203
2023-01-11T21:53:36.9885971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9886143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9886513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9886701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9887065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9887218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9887583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9887766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9888020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9888261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9888660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9889054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9889280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9889509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9889647Z skip: Skipped due to small world size. (4.258s)
2023-01-11T21:53:36.9889667Z 
2023-01-11T21:53:36.9889931Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9890041Z Ran 1 test in 4.259s
2023-01-11T21:53:36.9890111Z 
2023-01-11T21:53:36.9890262Z OK (skipped=1)
2023-01-11T21:53:36.9890283Z 
2023-01-11T21:53:36.9890408Z Generating XML reports...
2023-01-11T21:53:36.9890859Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213803.xml
2023-01-11T21:53:36.9891224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9891400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9891778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9891951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9891970Z 
2023-01-11T21:53:36.9892077Z Running tests...
2023-01-11T21:53:36.9892336Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9892650Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9892925Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9893142Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19307
2023-01-11T21:53:36.9893355Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19308
2023-01-11T21:53:36.9893722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9893878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9894252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9894437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9894799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9894978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9895347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9895531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9895775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9896019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9896400Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9896794Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9897025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9897259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9897360Z ok (4.976s)
2023-01-11T21:53:36.9897380Z 
2023-01-11T21:53:36.9897645Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9897756Z Ran 1 test in 4.976s
2023-01-11T21:53:36.9897775Z 
2023-01-11T21:53:36.9897864Z OK
2023-01-11T21:53:36.9897883Z 
2023-01-11T21:53:36.9897987Z Generating XML reports...
2023-01-11T21:53:36.9898431Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213810.xml
2023-01-11T21:53:36.9898796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9898969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9899346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9899642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9899663Z 
2023-01-11T21:53:36.9899774Z Running tests...
2023-01-11T21:53:36.9900036Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9900348Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9900606Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9900826Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19413
2023-01-11T21:53:36.9901041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19414
2023-01-11T21:53:36.9901407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9901580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9901961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9902148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9902506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9902659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9903027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9903211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9903456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9903697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9904102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9904497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9904725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9904949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9905032Z ok (4.347s)
2023-01-11T21:53:36.9905068Z 
2023-01-11T21:53:36.9905315Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9905424Z Ran 1 test in 4.348s
2023-01-11T21:53:36.9905443Z 
2023-01-11T21:53:36.9905533Z OK
2023-01-11T21:53:36.9905552Z 
2023-01-11T21:53:36.9905672Z Generating XML reports...
2023-01-11T21:53:36.9906119Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213817.xml
2023-01-11T21:53:36.9906494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9906669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9907044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9907216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9907235Z 
2023-01-11T21:53:36.9907342Z Running tests...
2023-01-11T21:53:36.9907603Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9907914Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9908201Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9908418Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19518
2023-01-11T21:53:36.9908727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19519
2023-01-11T21:53:36.9909102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9909258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9909632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9909818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9910181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9910353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9910718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9910911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9911156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9911400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9911782Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9912432Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9912663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9912891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9912991Z ok (5.268s)
2023-01-11T21:53:36.9913012Z 
2023-01-11T21:53:36.9913283Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9913395Z Ran 1 test in 5.268s
2023-01-11T21:53:36.9913414Z 
2023-01-11T21:53:36.9913506Z OK
2023-01-11T21:53:36.9913525Z 
2023-01-11T21:53:36.9913631Z Generating XML reports...
2023-01-11T21:53:36.9914079Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213824.xml
2023-01-11T21:53:36.9914443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9914614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9914989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9915177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9915196Z 
2023-01-11T21:53:36.9915302Z Running tests...
2023-01-11T21:53:36.9915568Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9915879Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9916139Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9916356Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19634
2023-01-11T21:53:36.9916571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19635
2023-01-11T21:53:36.9916937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9917109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9917483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9917670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9918166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9918327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9918697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9918885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9919131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9919368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9919765Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9920157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9920391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9920616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9920698Z ok (5.859s)
2023-01-11T21:53:36.9920735Z 
2023-01-11T21:53:36.9920980Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9921089Z Ran 1 test in 5.859s
2023-01-11T21:53:36.9921108Z 
2023-01-11T21:53:36.9921198Z OK
2023-01-11T21:53:36.9921217Z 
2023-01-11T21:53:36.9921338Z Generating XML reports...
2023-01-11T21:53:36.9921786Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213832.xml
2023-01-11T21:53:36.9922151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9922367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9922759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9922931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9922950Z 
2023-01-11T21:53:36.9923058Z Running tests...
2023-01-11T21:53:36.9923318Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9923632Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9923908Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9924128Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19749
2023-01-11T21:53:36.9924344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19750
2023-01-11T21:53:36.9924708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9924870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9925245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9925431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9925792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9925964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9926331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9926513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9926758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9927096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9927484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9927879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9928105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9928331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9928434Z ok (4.339s)
2023-01-11T21:53:36.9928454Z 
2023-01-11T21:53:36.9928711Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9928821Z Ran 1 test in 4.339s
2023-01-11T21:53:36.9928841Z 
2023-01-11T21:53:36.9928932Z OK
2023-01-11T21:53:36.9928951Z 
2023-01-11T21:53:36.9929054Z Generating XML reports...
2023-01-11T21:53:36.9929506Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213840.xml
2023-01-11T21:53:36.9929873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9930049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9930425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9930615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9930634Z 
2023-01-11T21:53:36.9930740Z Running tests...
2023-01-11T21:53:36.9930997Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9931308Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9931545Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:36.9931588Z 
2023-01-11T21:53:36.9931828Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9931938Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9931957Z 
2023-01-11T21:53:36.9932061Z OK (skipped=1)
2023-01-11T21:53:36.9932080Z 
2023-01-11T21:53:36.9932200Z Generating XML reports...
2023-01-11T21:53:36.9932646Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213847.xml
2023-01-11T21:53:36.9933013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9933186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9933561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9933733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9933756Z 
2023-01-11T21:53:36.9933871Z Running tests...
2023-01-11T21:53:36.9934132Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9934443Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9934700Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9935451Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.645s)
2023-01-11T21:53:36.9935471Z 
2023-01-11T21:53:36.9935725Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9935837Z Ran 1 test in 1.645s
2023-01-11T21:53:36.9935856Z 
2023-01-11T21:53:36.9936017Z OK (skipped=1)
2023-01-11T21:53:36.9936036Z 
2023-01-11T21:53:36.9936177Z Generating XML reports...
2023-01-11T21:53:36.9936633Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213849.xml
2023-01-11T21:53:36.9936999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9937172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9937548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9937739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9937758Z 
2023-01-11T21:53:36.9937865Z Running tests...
2023-01-11T21:53:36.9938124Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9938435Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9938694Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:36.9938731Z 
2023-01-11T21:53:36.9938970Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9939080Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9939099Z 
2023-01-11T21:53:36.9939204Z OK (skipped=1)
2023-01-11T21:53:36.9939222Z 
2023-01-11T21:53:36.9939341Z Generating XML reports...
2023-01-11T21:53:36.9939784Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213854.xml
2023-01-11T21:53:36.9940149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9940323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9940695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9940874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9940893Z 
2023-01-11T21:53:36.9940999Z Running tests...
2023-01-11T21:53:36.9941257Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9941565Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9941830Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:36.9941849Z 
2023-01-11T21:53:36.9942108Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9942218Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9942237Z 
2023-01-11T21:53:36.9942339Z OK (skipped=1)
2023-01-11T21:53:36.9942358Z 
2023-01-11T21:53:36.9942477Z Generating XML reports...
2023-01-11T21:53:36.9942907Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213856.xml
2023-01-11T21:53:36.9943283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9943455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9943830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9944019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9944039Z 
2023-01-11T21:53:36.9944146Z Running tests...
2023-01-11T21:53:36.9944404Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9944717Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9944967Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip: NCCL broadcast multigpu skipped (0.002s)
2023-01-11T21:53:36.9945004Z 
2023-01-11T21:53:36.9945245Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9945478Z Ran 1 test in 0.002s
2023-01-11T21:53:36.9945498Z 
2023-01-11T21:53:36.9945603Z OK (skipped=1)
2023-01-11T21:53:36.9945622Z 
2023-01-11T21:53:36.9945745Z Generating XML reports...
2023-01-11T21:53:36.9946193Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213858.xml
2023-01-11T21:53:36.9946560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9946730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9947107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9947279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9947298Z 
2023-01-11T21:53:36.9947404Z Running tests...
2023-01-11T21:53:36.9947667Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9947980Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9948246Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9948992Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.691s)
2023-01-11T21:53:36.9949012Z 
2023-01-11T21:53:36.9949271Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9949380Z Ran 1 test in 1.692s
2023-01-11T21:53:36.9949399Z 
2023-01-11T21:53:36.9949504Z OK (skipped=1)
2023-01-11T21:53:36.9949522Z 
2023-01-11T21:53:36.9949643Z Generating XML reports...
2023-01-11T21:53:36.9950079Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213901.xml
2023-01-11T21:53:36.9950445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9950617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9950992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9951177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9951196Z 
2023-01-11T21:53:36.9951300Z Running tests...
2023-01-11T21:53:36.9951559Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9952099Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9952415Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9953177Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85012 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.664s)
2023-01-11T21:53:36.9953218Z 
2023-01-11T21:53:36.9953456Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9953568Z Ran 1 test in 1.664s
2023-01-11T21:53:36.9953587Z 
2023-01-11T21:53:36.9953692Z OK (skipped=1)
2023-01-11T21:53:36.9953711Z 
2023-01-11T21:53:36.9953834Z Generating XML reports...
2023-01-11T21:53:36.9954274Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213905.xml
2023-01-11T21:53:36.9954643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9954949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9955338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9955520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9955539Z 
2023-01-11T21:53:36.9955647Z Running tests...
2023-01-11T21:53:36.9955906Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9956216Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9956535Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9957279Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85339 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.670s)
2023-01-11T21:53:36.9957302Z 
2023-01-11T21:53:36.9957560Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9957670Z Ran 1 test in 1.670s
2023-01-11T21:53:36.9957689Z 
2023-01-11T21:53:36.9957795Z OK (skipped=1)
2023-01-11T21:53:36.9957813Z 
2023-01-11T21:53:36.9957934Z Generating XML reports...
2023-01-11T21:53:36.9958363Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213909.xml
2023-01-11T21:53:36.9958732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9958905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9959281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9959478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9959497Z 
2023-01-11T21:53:36.9959601Z Running tests...
2023-01-11T21:53:36.9959857Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9960168Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9960423Z   test_ddp_apply_optim_in_backward (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9960640Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20122
2023-01-11T21:53:36.9960856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20123
2023-01-11T21:53:36.9961224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9961397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9961778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9961967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9962326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9962495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9962847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9963034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9963280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9963527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9963973Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9964431Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9964664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9964895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9965673Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T21:53:36.9965769Z   warnings.warn(
2023-01-11T21:53:36.9966547Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T21:53:36.9966659Z   warnings.warn(
2023-01-11T21:53:36.9966897Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9967127Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9967360Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9967592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9967826Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9968047Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9968253Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9968491Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9968590Z ok (6.982s)
2023-01-11T21:53:36.9968610Z 
2023-01-11T21:53:36.9968873Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9968981Z Ran 1 test in 6.982s
2023-01-11T21:53:36.9969000Z 
2023-01-11T21:53:36.9969089Z OK
2023-01-11T21:53:36.9969107Z 
2023-01-11T21:53:36.9969228Z Generating XML reports...
2023-01-11T21:53:36.9969682Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213913.xml
2023-01-11T21:53:36.9970050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9970209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9970585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9970782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9970801Z 
2023-01-11T21:53:36.9970909Z Running tests...
2023-01-11T21:53:36.9971167Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9971481Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9971790Z   test_ddp_apply_optim_in_backward_grad_as_bucket_view_false (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9972011Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20240
2023-01-11T21:53:36.9972210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20241
2023-01-11T21:53:36.9972578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9972806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9973227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9973422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9973784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9973957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9974328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9974513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9974744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9974989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9975395Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9975793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9976024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9976253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9977033Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T21:53:36.9977145Z   warnings.warn(
2023-01-11T21:53:36.9977923Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T21:53:36.9978037Z   warnings.warn(
2023-01-11T21:53:36.9978256Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9978483Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9978717Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9978946Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9979046Z ok (6.370s)
2023-01-11T21:53:36.9979066Z 
2023-01-11T21:53:36.9979329Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9979443Z Ran 1 test in 6.370s
2023-01-11T21:53:36.9979462Z 
2023-01-11T21:53:36.9979555Z OK
2023-01-11T21:53:36.9979574Z 
2023-01-11T21:53:36.9979678Z Generating XML reports...
2023-01-11T21:53:36.9980130Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213923.xml
2023-01-11T21:53:36.9980499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9980675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9981051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9981241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9981260Z 
2023-01-11T21:53:36.9981365Z Running tests...
2023-01-11T21:53:36.9981626Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9982036Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9982319Z   test_ddp_apply_optim_in_backward_ignored_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9982539Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20358
2023-01-11T21:53:36.9982755Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20359
2023-01-11T21:53:36.9983123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9983298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9983673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9983860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9984224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9984381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9984749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9984934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9985181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9985423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9985824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9986219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9986451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9986681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9987462Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T21:53:36.9987556Z   warnings.warn(
2023-01-11T21:53:36.9988332Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T21:53:36.9988440Z   warnings.warn(
2023-01-11T21:53:36.9988544Z ok (6.284s)
2023-01-11T21:53:36.9988564Z 
2023-01-11T21:53:36.9988828Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9988938Z Ran 1 test in 6.285s
2023-01-11T21:53:36.9988957Z 
2023-01-11T21:53:36.9989046Z OK
2023-01-11T21:53:36.9989065Z 
2023-01-11T21:53:36.9989186Z Generating XML reports...
2023-01-11T21:53:36.9989634Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213932.xml
2023-01-11T21:53:36.9989981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9990156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9990531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9990718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9990786Z 
2023-01-11T21:53:36.9990902Z Running tests...
2023-01-11T21:53:36.9991204Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9991528Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:36.9991799Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:36.9992246Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20478
2023-01-11T21:53:36.9992473Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20479
2023-01-11T21:53:36.9992851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9993023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9993396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9993591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9993951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9994120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9994490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9994658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9994901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:36.9995144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:36.9995544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9995941Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:36.9996174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:36.9996400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:36.9996636Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9996871Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:36.9996955Z ok (5.635s)
2023-01-11T21:53:36.9996976Z 
2023-01-11T21:53:36.9997239Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9997348Z Ran 1 test in 5.635s
2023-01-11T21:53:36.9997367Z 
2023-01-11T21:53:36.9997455Z OK
2023-01-11T21:53:36.9997474Z 
2023-01-11T21:53:36.9997596Z Generating XML reports...
2023-01-11T21:53:36.9998048Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213941.xml
2023-01-11T21:53:36.9998421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:36.9998593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:36.9998953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:36.9999143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:36.9999162Z 
2023-01-11T21:53:36.9999267Z Running tests...
2023-01-11T21:53:36.9999526Z ----------------------------------------------------------------------
2023-01-11T21:53:36.9999839Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0000116Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0000476Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20596
2023-01-11T21:53:37.0000699Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20597
2023-01-11T21:53:37.0001070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0001224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0001599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0001787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0002145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0002319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0002689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0002880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0003124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0003352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0003752Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0004144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0004372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0004598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0004834Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0005076Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0005304Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0005537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0005619Z ok (5.671s)
2023-01-11T21:53:37.0005639Z 
2023-01-11T21:53:37.0005901Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0006010Z Ran 1 test in 5.671s
2023-01-11T21:53:37.0006029Z 
2023-01-11T21:53:37.0006120Z OK
2023-01-11T21:53:37.0006139Z 
2023-01-11T21:53:37.0006260Z Generating XML reports...
2023-01-11T21:53:37.0006707Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213949.xml
2023-01-11T21:53:37.0007070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0007250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0007612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0007802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0007821Z 
2023-01-11T21:53:37.0007928Z Running tests...
2023-01-11T21:53:37.0008187Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0008497Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0008770Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0009562Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.658s)
2023-01-11T21:53:37.0009620Z 
2023-01-11T21:53:37.0009890Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0010001Z Ran 1 test in 1.658s
2023-01-11T21:53:37.0010021Z 
2023-01-11T21:53:37.0010128Z OK (skipped=1)
2023-01-11T21:53:37.0010147Z 
2023-01-11T21:53:37.0010250Z Generating XML reports...
2023-01-11T21:53:37.0010700Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213957.xml
2023-01-11T21:53:37.0011068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0011240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0011617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0011814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0011833Z 
2023-01-11T21:53:37.0011939Z Running tests...
2023-01-11T21:53:37.0012197Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0012492Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0012783Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0013531Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.698s)
2023-01-11T21:53:37.0013551Z 
2023-01-11T21:53:37.0013809Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0013922Z Ran 1 test in 1.698s
2023-01-11T21:53:37.0013942Z 
2023-01-11T21:53:37.0014050Z OK (skipped=1)
2023-01-11T21:53:37.0014069Z 
2023-01-11T21:53:37.0014191Z Generating XML reports...
2023-01-11T21:53:37.0014637Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214001.xml
2023-01-11T21:53:37.0015004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0015176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0015535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0015724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0015742Z 
2023-01-11T21:53:37.0015848Z Running tests...
2023-01-11T21:53:37.0016105Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0016424Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0016714Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0016933Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20782
2023-01-11T21:53:37.0017150Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20783
2023-01-11T21:53:37.0017514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0017671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0018047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0018235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0018650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0018873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0019245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0019434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0019680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0019907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0020305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0020699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0020935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0021163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0021369Z <class 'torch.nn.parameter.Parameter'>
2023-01-11T21:53:37.0021470Z ok (5.564s)
2023-01-11T21:53:37.0021490Z 
2023-01-11T21:53:37.0021751Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0021861Z Ran 1 test in 5.564s
2023-01-11T21:53:37.0021881Z 
2023-01-11T21:53:37.0021953Z OK
2023-01-11T21:53:37.0021971Z 
2023-01-11T21:53:37.0022092Z Generating XML reports...
2023-01-11T21:53:37.0022587Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214005.xml
2023-01-11T21:53:37.0022962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0023136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0023521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0023711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0023731Z 
2023-01-11T21:53:37.0023837Z Running tests...
2023-01-11T21:53:37.0024079Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0024395Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0024706Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0024925Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20897
2023-01-11T21:53:37.0025139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20898
2023-01-11T21:53:37.0025510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0025689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0026062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0026252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0026597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0026768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0027135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0027321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0027565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0027919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0028327Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0028720Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0028949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0029160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0029260Z ok (5.651s)
2023-01-11T21:53:37.0029280Z 
2023-01-11T21:53:37.0029544Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0029654Z Ran 1 test in 5.651s
2023-01-11T21:53:37.0029673Z 
2023-01-11T21:53:37.0029763Z OK
2023-01-11T21:53:37.0029782Z 
2023-01-11T21:53:37.0029908Z Generating XML reports...
2023-01-11T21:53:37.0030359Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214013.xml
2023-01-11T21:53:37.0030728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0030884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0031260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0031449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0031468Z 
2023-01-11T21:53:37.0031574Z Running tests...
2023-01-11T21:53:37.0032100Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0032450Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0032722Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0032947Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21012
2023-01-11T21:53:37.0033162Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21013
2023-01-11T21:53:37.0033516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0033690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0034065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0034253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0034614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0034784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0035161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0035348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0035578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0035823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0036225Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0036617Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0036846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0037073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0037444Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0037688Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0037790Z ok (6.041s)
2023-01-11T21:53:37.0037809Z 
2023-01-11T21:53:37.0038057Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0038168Z Ran 1 test in 6.041s
2023-01-11T21:53:37.0038188Z 
2023-01-11T21:53:37.0038280Z OK
2023-01-11T21:53:37.0038299Z 
2023-01-11T21:53:37.0038421Z Generating XML reports...
2023-01-11T21:53:37.0038869Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214022.xml
2023-01-11T21:53:37.0039236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0039411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0039791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0039963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0040000Z 
2023-01-11T21:53:37.0040089Z Running tests...
2023-01-11T21:53:37.0040349Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0040663Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0040958Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0041178Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21131
2023-01-11T21:53:37.0041393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21132
2023-01-11T21:53:37.0041757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0041936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0042295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0042483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0042842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0043012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0043382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0043569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0043811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0044062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0044461Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0044835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0045065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0045293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0046125Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T21:53:37.0046271Z ok (5.687s)
2023-01-11T21:53:37.0046291Z 
2023-01-11T21:53:37.0046554Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0046667Z Ran 1 test in 5.687s
2023-01-11T21:53:37.0046686Z 
2023-01-11T21:53:37.0046777Z OK
2023-01-11T21:53:37.0046796Z 
2023-01-11T21:53:37.0046921Z Generating XML reports...
2023-01-11T21:53:37.0047368Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214030.xml
2023-01-11T21:53:37.0047718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0047893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0048273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0048465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0048484Z 
2023-01-11T21:53:37.0048591Z Running tests...
2023-01-11T21:53:37.0048848Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0049161Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0049444Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0050189Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78235 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.633s)
2023-01-11T21:53:37.0050210Z 
2023-01-11T21:53:37.0050473Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0050568Z Ran 1 test in 1.633s
2023-01-11T21:53:37.0050587Z 
2023-01-11T21:53:37.0050695Z OK (skipped=1)
2023-01-11T21:53:37.0050714Z 
2023-01-11T21:53:37.0050837Z Generating XML reports...
2023-01-11T21:53:37.0051284Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214038.xml
2023-01-11T21:53:37.0051651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0051825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0052198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0052384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0052403Z 
2023-01-11T21:53:37.0052490Z Running tests...
2023-01-11T21:53:37.0052757Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0053069Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0053394Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip: Gloo-only test (0.002s)
2023-01-11T21:53:37.0053413Z 
2023-01-11T21:53:37.0053666Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0053775Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0053794Z 
2023-01-11T21:53:37.0053898Z OK (skipped=1)
2023-01-11T21:53:37.0053916Z 
2023-01-11T21:53:37.0054038Z Generating XML reports...
2023-01-11T21:53:37.0054484Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214043.xml
2023-01-11T21:53:37.0054835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0055009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0055485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0055682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0055702Z 
2023-01-11T21:53:37.0055809Z Running tests...
2023-01-11T21:53:37.0056069Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0056383Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0056635Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0057374Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77324 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.669s)
2023-01-11T21:53:37.0057398Z 
2023-01-11T21:53:37.0057659Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0057753Z Ran 1 test in 1.669s
2023-01-11T21:53:37.0057772Z 
2023-01-11T21:53:37.0057881Z OK (skipped=1)
2023-01-11T21:53:37.0057900Z 
2023-01-11T21:53:37.0058022Z Generating XML reports...
2023-01-11T21:53:37.0058467Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214045.xml
2023-01-11T21:53:37.0058837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0059012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0059388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0059579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0059602Z 
2023-01-11T21:53:37.0059692Z Running tests...
2023-01-11T21:53:37.0059954Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0060264Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0060534Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0060753Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21350
2023-01-11T21:53:37.0060970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21351
2023-01-11T21:53:37.0061337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0061508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0061880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0062057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0062416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0062587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0062960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0063149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0063395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0063640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0064039Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0064474Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0064756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0064987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0065777Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1331: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2023-01-11T21:53:37.0066106Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2023-01-11T21:53:37.0066890Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1331: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2023-01-11T21:53:37.0067219Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2023-01-11T21:53:37.0067320Z ok (6.197s)
2023-01-11T21:53:37.0067340Z 
2023-01-11T21:53:37.0067603Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0067714Z Ran 1 test in 6.198s
2023-01-11T21:53:37.0067733Z 
2023-01-11T21:53:37.0067822Z OK
2023-01-11T21:53:37.0067841Z 
2023-01-11T21:53:37.0067945Z Generating XML reports...
2023-01-11T21:53:37.0068399Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214049.xml
2023-01-11T21:53:37.0068766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0068945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0069326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0069515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0069534Z 
2023-01-11T21:53:37.0069640Z Running tests...
2023-01-11T21:53:37.0069900Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0070195Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0070470Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0071216Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.689s)
2023-01-11T21:53:37.0071243Z 
2023-01-11T21:53:37.0071506Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0071616Z Ran 1 test in 1.689s
2023-01-11T21:53:37.0071635Z 
2023-01-11T21:53:37.0071742Z OK (skipped=1)
2023-01-11T21:53:37.0071761Z 
2023-01-11T21:53:37.0072132Z Generating XML reports...
2023-01-11T21:53:37.0072598Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214058.xml
2023-01-11T21:53:37.0072967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0073140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0073502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0073693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0073795Z 
2023-01-11T21:53:37.0073960Z Running tests...
2023-01-11T21:53:37.0074231Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0074546Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0074823Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0075565Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.675s)
2023-01-11T21:53:37.0075585Z 
2023-01-11T21:53:37.0075843Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0075952Z Ran 1 test in 1.675s
2023-01-11T21:53:37.0075971Z 
2023-01-11T21:53:37.0076081Z OK (skipped=1)
2023-01-11T21:53:37.0076100Z 
2023-01-11T21:53:37.0076207Z Generating XML reports...
2023-01-11T21:53:37.0076657Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214102.xml
2023-01-11T21:53:37.0077021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0077194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0077574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0077763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0077782Z 
2023-01-11T21:53:37.0077888Z Running tests...
2023-01-11T21:53:37.0078147Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0078441Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0078743Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0078966Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21567
2023-01-11T21:53:37.0079185Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21568
2023-01-11T21:53:37.0079552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0079724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0080102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0080290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0080651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0080813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0081185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0081372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0081616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0081862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0082261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0082658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0082887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0083249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:53:37.0083461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0083698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:53:37.0084101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:37.0084490Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:37.0084727Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0084964Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0085199Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0085439Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0085522Z ok (6.487s)
2023-01-11T21:53:37.0085559Z 
2023-01-11T21:53:37.0085808Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0085918Z Ran 1 test in 6.487s
2023-01-11T21:53:37.0085937Z 
2023-01-11T21:53:37.0086027Z OK
2023-01-11T21:53:37.0086047Z 
2023-01-11T21:53:37.0086172Z Generating XML reports...
2023-01-11T21:53:37.0086622Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214106.xml
2023-01-11T21:53:37.0086990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0087165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0087543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0087725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0087762Z 
2023-01-11T21:53:37.0087851Z Running tests...
2023-01-11T21:53:37.0088111Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0088426Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0088705Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0088923Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21688
2023-01-11T21:53:37.0089139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21689
2023-01-11T21:53:37.0089505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0089660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0090043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0090233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0090596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0090767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0091137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0091325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0091574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0091819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0092250Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0092711Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0092942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0093224Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2023-01-11T21:53:37.0093449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0093720Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2023-01-11T21:53:37.0093959Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0094188Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0094430Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0094646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0094927Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2023-01-11T21:53:37.0095200Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2023-01-11T21:53:37.0095474Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2023-01-11T21:53:37.0095746Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2023-01-11T21:53:37.0095978Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0096212Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0096449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0096679Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0096935Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2023-01-11T21:53:37.0097208Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2023-01-11T21:53:37.0097485Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2023-01-11T21:53:37.0097756Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2023-01-11T21:53:37.0097989Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0098221Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0098458Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0098692Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0098774Z ok (6.907s)
2023-01-11T21:53:37.0098811Z 
2023-01-11T21:53:37.0099059Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0099171Z Ran 1 test in 6.907s
2023-01-11T21:53:37.0099190Z 
2023-01-11T21:53:37.0099281Z OK
2023-01-11T21:53:37.0099300Z 
2023-01-11T21:53:37.0099422Z Generating XML reports...
2023-01-11T21:53:37.0099872Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214115.xml
2023-01-11T21:53:37.0100239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0100412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0100884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0101060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0101099Z 
2023-01-11T21:53:37.0101188Z Running tests...
2023-01-11T21:53:37.0101449Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0101763Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0102037Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0102779Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.681s)
2023-01-11T21:53:37.0102803Z 
2023-01-11T21:53:37.0103065Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0103176Z Ran 1 test in 1.681s
2023-01-11T21:53:37.0103196Z 
2023-01-11T21:53:37.0103301Z OK (skipped=1)
2023-01-11T21:53:37.0103320Z 
2023-01-11T21:53:37.0103443Z Generating XML reports...
2023-01-11T21:53:37.0103870Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214125.xml
2023-01-11T21:53:37.0104240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0104415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0104791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0104984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0105007Z 
2023-01-11T21:53:37.0105114Z Running tests...
2023-01-11T21:53:37.0105380Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0105690Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0105947Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0106165Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21841
2023-01-11T21:53:37.0106381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21842
2023-01-11T21:53:37.0106747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0106920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0107298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0107493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0107853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0108025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0108378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0108566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0108816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0109063Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0109462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0109905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0110182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0110734Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:53:37.0110962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0111501Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:53:37.0111744Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0112140Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0112429Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2023-01-11T21:53:37.0112704Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2023-01-11T21:53:37.0113005Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2023-01-11T21:53:37.0113300Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2023-01-11T21:53:37.0113630Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2023-01-11T21:53:37.0113963Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2023-01-11T21:53:37.0114289Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2023-01-11T21:53:37.0114613Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2023-01-11T21:53:37.0114850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0115067Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0115167Z ok (6.201s)
2023-01-11T21:53:37.0115188Z 
2023-01-11T21:53:37.0115469Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0115586Z Ran 1 test in 6.202s
2023-01-11T21:53:37.0115605Z 
2023-01-11T21:53:37.0115702Z OK
2023-01-11T21:53:37.0115720Z 
2023-01-11T21:53:37.0115842Z Generating XML reports...
2023-01-11T21:53:37.0116300Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214129.xml
2023-01-11T21:53:37.0116671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0116830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0117207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0117398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0117418Z 
2023-01-11T21:53:37.0117525Z Running tests...
2023-01-11T21:53:37.0117786Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0118240Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0118645Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T21:53:37.0118667Z 
2023-01-11T21:53:37.0118929Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0119040Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0119059Z 
2023-01-11T21:53:37.0119147Z OK (skipped=1)
2023-01-11T21:53:37.0119187Z 
2023-01-11T21:53:37.0119291Z Generating XML reports...
2023-01-11T21:53:37.0119740Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214138.xml
2023-01-11T21:53:37.0120111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0120283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0120670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0120862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0120881Z 
2023-01-11T21:53:37.0120989Z Running tests...
2023-01-11T21:53:37.0121247Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0121543Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0121932Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T21:53:37.0121952Z 
2023-01-11T21:53:37.0122210Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0122318Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0122341Z 
2023-01-11T21:53:37.0122490Z OK (skipped=1)
2023-01-11T21:53:37.0122512Z 
2023-01-11T21:53:37.0122636Z Generating XML reports...
2023-01-11T21:53:37.0123084Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214140.xml
2023-01-11T21:53:37.0123453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0123627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0123987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0124176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0124195Z 
2023-01-11T21:53:37.0124300Z Running tests...
2023-01-11T21:53:37.0124560Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0124877Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0125387Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T21:53:37.0125408Z 
2023-01-11T21:53:37.0125670Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0125782Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0125801Z 
2023-01-11T21:53:37.0125907Z OK (skipped=1)
2023-01-11T21:53:37.0125926Z 
2023-01-11T21:53:37.0126030Z Generating XML reports...
2023-01-11T21:53:37.0126483Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214143.xml
2023-01-11T21:53:37.0126851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0127025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0127501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0127697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0127716Z 
2023-01-11T21:53:37.0127823Z Running tests...
2023-01-11T21:53:37.0128085Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0128395Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0128824Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T21:53:37.0128863Z 
2023-01-11T21:53:37.0129105Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0129218Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0129237Z 
2023-01-11T21:53:37.0129344Z OK (skipped=1)
2023-01-11T21:53:37.0129363Z 
2023-01-11T21:53:37.0129484Z Generating XML reports...
2023-01-11T21:53:37.0129932Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214145.xml
2023-01-11T21:53:37.0130299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0130473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0130848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0131020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0131056Z 
2023-01-11T21:53:37.0131145Z Running tests...
2023-01-11T21:53:37.0131402Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0131721Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0132165Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T21:53:37.0132184Z 
2023-01-11T21:53:37.0132450Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0132559Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0132578Z 
2023-01-11T21:53:37.0132684Z OK (skipped=1)
2023-01-11T21:53:37.0132702Z 
2023-01-11T21:53:37.0132824Z Generating XML reports...
2023-01-11T21:53:37.0133270Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214147.xml
2023-01-11T21:53:37.0133621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0133803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0134179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0134367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0134387Z 
2023-01-11T21:53:37.0134492Z Running tests...
2023-01-11T21:53:37.0134750Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0135062Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0135503Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T21:53:37.0135523Z 
2023-01-11T21:53:37.0135780Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0135928Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0135983Z 
2023-01-11T21:53:37.0136095Z OK (skipped=1)
2023-01-11T21:53:37.0136113Z 
2023-01-11T21:53:37.0136235Z Generating XML reports...
2023-01-11T21:53:37.0136685Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214150.xml
2023-01-11T21:53:37.0137050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0137224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0137601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0137790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0137809Z 
2023-01-11T21:53:37.0137916Z Running tests...
2023-01-11T21:53:37.0138159Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0138479Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0138923Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T21:53:37.0138942Z 
2023-01-11T21:53:37.0139201Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0139311Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0139330Z 
2023-01-11T21:53:37.0139435Z OK (skipped=1)
2023-01-11T21:53:37.0139454Z 
2023-01-11T21:53:37.0139577Z Generating XML reports...
2023-01-11T21:53:37.0140021Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214152.xml
2023-01-11T21:53:37.0140388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0140551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0140928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0141117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0141136Z 
2023-01-11T21:53:37.0141242Z Running tests...
2023-01-11T21:53:37.0141502Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0141812Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0142253Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T21:53:37.0142273Z 
2023-01-11T21:53:37.0142536Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0142650Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0142669Z 
2023-01-11T21:53:37.0142757Z OK (skipped=1)
2023-01-11T21:53:37.0142795Z 
2023-01-11T21:53:37.0142900Z Generating XML reports...
2023-01-11T21:53:37.0143344Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214155.xml
2023-01-11T21:53:37.0143711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0143884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0144261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0144450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0144470Z 
2023-01-11T21:53:37.0144576Z Running tests...
2023-01-11T21:53:37.0144929Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0145229Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0145673Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T21:53:37.0145693Z 
2023-01-11T21:53:37.0145951Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0146059Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0146081Z 
2023-01-11T21:53:37.0146185Z OK (skipped=1)
2023-01-11T21:53:37.0146204Z 
2023-01-11T21:53:37.0146328Z Generating XML reports...
2023-01-11T21:53:37.0146774Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214157.xml
2023-01-11T21:53:37.0147146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0147322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0147679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0147869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0147888Z 
2023-01-11T21:53:37.0147994Z Running tests...
2023-01-11T21:53:37.0148251Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0148563Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0149003Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T21:53:37.0149026Z 
2023-01-11T21:53:37.0149288Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0149398Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0149417Z 
2023-01-11T21:53:37.0149523Z OK (skipped=1)
2023-01-11T21:53:37.0149542Z 
2023-01-11T21:53:37.0149664Z Generating XML reports...
2023-01-11T21:53:37.0150092Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214159.xml
2023-01-11T21:53:37.0150460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0150635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0151012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0151201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0151220Z 
2023-01-11T21:53:37.0151332Z Running tests...
2023-01-11T21:53:37.0151594Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0152087Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0152475Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T21:53:37.0152516Z 
2023-01-11T21:53:37.0152768Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0152880Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0152899Z 
2023-01-11T21:53:37.0153006Z OK (skipped=1)
2023-01-11T21:53:37.0153024Z 
2023-01-11T21:53:37.0153147Z Generating XML reports...
2023-01-11T21:53:37.0153592Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214202.xml
2023-01-11T21:53:37.0154033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0154272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0154654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0154827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0154866Z 
2023-01-11T21:53:37.0154955Z Running tests...
2023-01-11T21:53:37.0155214Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0155526Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0155914Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T21:53:37.0155934Z 
2023-01-11T21:53:37.0156195Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0156309Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0156328Z 
2023-01-11T21:53:37.0156435Z OK (skipped=1)
2023-01-11T21:53:37.0156453Z 
2023-01-11T21:53:37.0156573Z Generating XML reports...
2023-01-11T21:53:37.0157000Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214204.xml
2023-01-11T21:53:37.0157370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0157544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0157918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0158106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0158125Z 
2023-01-11T21:53:37.0158231Z Running tests...
2023-01-11T21:53:37.0158496Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0158808Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0159078Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0159800Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.687s)
2023-01-11T21:53:37.0159839Z 
2023-01-11T21:53:37.0160079Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0160189Z Ran 1 test in 1.688s
2023-01-11T21:53:37.0160208Z 
2023-01-11T21:53:37.0160312Z OK (skipped=1)
2023-01-11T21:53:37.0160331Z 
2023-01-11T21:53:37.0160454Z Generating XML reports...
2023-01-11T21:53:37.0160906Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214207.xml
2023-01-11T21:53:37.0161278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0161450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0161828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0162017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0162037Z 
2023-01-11T21:53:37.0162125Z Running tests...
2023-01-11T21:53:37.0162388Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0162697Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0162954Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0163266Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22390
2023-01-11T21:53:37.0163487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22391
2023-01-11T21:53:37.0163858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0164031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0164388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0164578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0164941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0165113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0165487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0165677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0165923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0166167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0166568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0166949Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0167179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0167406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0167510Z ok (5.773s)
2023-01-11T21:53:37.0167530Z 
2023-01-11T21:53:37.0167795Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0167904Z Ran 1 test in 5.773s
2023-01-11T21:53:37.0167924Z 
2023-01-11T21:53:37.0168017Z OK
2023-01-11T21:53:37.0168036Z 
2023-01-11T21:53:37.0168158Z Generating XML reports...
2023-01-11T21:53:37.0168587Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214211.xml
2023-01-11T21:53:37.0168956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0169128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0169506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0169696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0169716Z 
2023-01-11T21:53:37.0169827Z Running tests...
2023-01-11T21:53:37.0170086Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0170399Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0170674Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0170877Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22504
2023-01-11T21:53:37.0171094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22505
2023-01-11T21:53:37.0171460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0171633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0172012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0172294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0172663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0172834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0173189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0173377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0173621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0173865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0174262Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0174658Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0174892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0175119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0175357Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0175576Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0175980Z /opt/conda/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmprdv_6kl_'>
2023-01-11T21:53:37.0176145Z   _warnings.warn(warn_message, ResourceWarning)
2023-01-11T21:53:37.0176534Z /opt/conda/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpze7e0y8v'>
2023-01-11T21:53:37.0176697Z   _warnings.warn(warn_message, ResourceWarning)
2023-01-11T21:53:37.0176800Z ok (6.159s)
2023-01-11T21:53:37.0176823Z 
2023-01-11T21:53:37.0177085Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0177196Z Ran 1 test in 6.159s
2023-01-11T21:53:37.0177215Z 
2023-01-11T21:53:37.0177288Z OK
2023-01-11T21:53:37.0177324Z 
2023-01-11T21:53:37.0177430Z Generating XML reports...
2023-01-11T21:53:37.0177878Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214219.xml
2023-01-11T21:53:37.0178243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0178416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0178791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0178980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0179003Z 
2023-01-11T21:53:37.0179116Z Running tests...
2023-01-11T21:53:37.0179374Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0179672Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0179946Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.005s)
2023-01-11T21:53:37.0179965Z 
2023-01-11T21:53:37.0180224Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0180333Z Ran 1 test in 0.005s
2023-01-11T21:53:37.0180353Z 
2023-01-11T21:53:37.0180460Z OK (skipped=1)
2023-01-11T21:53:37.0180479Z 
2023-01-11T21:53:37.0180600Z Generating XML reports...
2023-01-11T21:53:37.0181044Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214228.xml
2023-01-11T21:53:37.0181459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0181688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0182050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0182241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0182260Z 
2023-01-11T21:53:37.0182366Z Running tests...
2023-01-11T21:53:37.0182625Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0182938Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0183210Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0183432Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22656
2023-01-11T21:53:37.0183648Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22657
2023-01-11T21:53:37.0184002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0184177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0184555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0184746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0185109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0185282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0185653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0185837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0186091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0186319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0186717Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0187113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0187342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0187571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0187809Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0188044Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0188148Z ok (6.191s)
2023-01-11T21:53:37.0188171Z 
2023-01-11T21:53:37.0188440Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0188532Z Ran 1 test in 6.192s
2023-01-11T21:53:37.0188551Z 
2023-01-11T21:53:37.0188641Z OK
2023-01-11T21:53:37.0188660Z 
2023-01-11T21:53:37.0188782Z Generating XML reports...
2023-01-11T21:53:37.0189230Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214230.xml
2023-01-11T21:53:37.0189600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0189775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0190151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0190341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0190410Z 
2023-01-11T21:53:37.0190504Z Running tests...
2023-01-11T21:53:37.0190804Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0191123Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0191409Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0191628Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22775
2023-01-11T21:53:37.0192018Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22776
2023-01-11T21:53:37.0192410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0192584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0192962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0193142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0193507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0193677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0194048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0194235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0194482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0194729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0195126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0195512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0195746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0195972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0196211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:53:37.0196451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:53:37.0196849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:37.0197241Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:37.0197481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:53:37.0197726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:53:37.0198097Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:53:37.0198492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:53:37.0198595Z ok (5.247s)
2023-01-11T21:53:37.0198615Z 
2023-01-11T21:53:37.0198876Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0198986Z Ran 1 test in 5.248s
2023-01-11T21:53:37.0199005Z 
2023-01-11T21:53:37.0199099Z OK
2023-01-11T21:53:37.0199118Z 
2023-01-11T21:53:37.0199239Z Generating XML reports...
2023-01-11T21:53:37.0199689Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214239.xml
2023-01-11T21:53:37.0200201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0200366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0200745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0200934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0200954Z 
2023-01-11T21:53:37.0201064Z Running tests...
2023-01-11T21:53:37.0201324Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0201636Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0201919Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0202136Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22897
2023-01-11T21:53:37.0202343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22898
2023-01-11T21:53:37.0202711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0202886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0203262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0203452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0203813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0203986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0204359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0204552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0204781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0205026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0205423Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0205816Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0206050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0206276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0206515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:53:37.0206762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:53:37.0207161Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:37.0207535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:37.0207775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:53:37.0208012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:53:37.0208402Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:53:37.0208797Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:53:37.0209342Z [W ProcessGroupNCCL.cpp:950] [Rank 0] Found key in store: NCCLABORTEDCOMM:20de13ac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2023-01-11T21:53:37.0209868Z [W ProcessGroupNCCL.cpp:950] [Rank 1] Found key in store: NCCLABORTEDCOMM:20de13ac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2023-01-11T21:53:37.0209973Z ok (15.515s)
2023-01-11T21:53:37.0209993Z 
2023-01-11T21:53:37.0210261Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0210372Z Ran 1 test in 15.516s
2023-01-11T21:53:37.0210392Z 
2023-01-11T21:53:37.0210484Z OK
2023-01-11T21:53:37.0210506Z 
2023-01-11T21:53:37.0210614Z Generating XML reports...
2023-01-11T21:53:37.0211064Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214247.xml
2023-01-11T21:53:37.0211428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0211602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0211983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0212172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0212191Z 
2023-01-11T21:53:37.0212298Z Running tests...
2023-01-11T21:53:37.0212562Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0212878Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0213179Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0213399Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23019
2023-01-11T21:53:37.0213613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23020
2023-01-11T21:53:37.0213982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0214154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0214531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0214720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0215082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0215243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0215617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0215801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0216046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0216290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0216690Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0217086Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0217317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0217646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0217735Z ok (6.284s)
2023-01-11T21:53:37.0217755Z 
2023-01-11T21:53:37.0218019Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0218128Z Ran 1 test in 6.285s
2023-01-11T21:53:37.0218148Z 
2023-01-11T21:53:37.0218239Z OK
2023-01-11T21:53:37.0218257Z 
2023-01-11T21:53:37.0218378Z Generating XML reports...
2023-01-11T21:53:37.0218829Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214305.xml
2023-01-11T21:53:37.0219198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0219371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0219747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0219926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0219945Z 
2023-01-11T21:53:37.0220053Z Running tests...
2023-01-11T21:53:37.0220313Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0220625Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0220919Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0221140Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23138
2023-01-11T21:53:37.0221354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23139
2023-01-11T21:53:37.0221723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0221876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0222260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0222492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0222860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0223031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0223403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0223591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0223834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0224077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0224462Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0224866Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0225096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0225324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0225428Z ok (6.373s)
2023-01-11T21:53:37.0225449Z 
2023-01-11T21:53:37.0225706Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0225815Z Ran 1 test in 6.374s
2023-01-11T21:53:37.0225834Z 
2023-01-11T21:53:37.0225924Z OK
2023-01-11T21:53:37.0225943Z 
2023-01-11T21:53:37.0226047Z Generating XML reports...
2023-01-11T21:53:37.0226495Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214313.xml
2023-01-11T21:53:37.0226991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0227171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0227546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0227738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0227758Z 
2023-01-11T21:53:37.0227866Z Running tests...
2023-01-11T21:53:37.0228123Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0228436Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0228679Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0228900Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23257
2023-01-11T21:53:37.0229125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23258
2023-01-11T21:53:37.0229488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0229664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0230040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0230231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0230591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0230745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0231118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0231303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0231554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0231800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0232448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0232847Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0233077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0233307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0233389Z ok (6.229s)
2023-01-11T21:53:37.0233410Z 
2023-01-11T21:53:37.0233674Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0233789Z Ran 1 test in 6.229s
2023-01-11T21:53:37.0233812Z 
2023-01-11T21:53:37.0233976Z OK
2023-01-11T21:53:37.0233995Z 
2023-01-11T21:53:37.0234117Z Generating XML reports...
2023-01-11T21:53:37.0234569Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214322.xml
2023-01-11T21:53:37.0234936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0235109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0235485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0235658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0235677Z 
2023-01-11T21:53:37.0235783Z Running tests...
2023-01-11T21:53:37.0236047Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0236575Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0237063Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0237342Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23372
2023-01-11T21:53:37.0237561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23373
2023-01-11T21:53:37.0237941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0238097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0238478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0238665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0239030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0239208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0239579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0239763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0240012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0240257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0240640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0241037Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0241273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0241499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0242275Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T21:53:37.0243057Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T21:53:37.0243304Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0243538Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0243639Z ok (6.079s)
2023-01-11T21:53:37.0243660Z 
2023-01-11T21:53:37.0243925Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0244033Z Ran 1 test in 6.080s
2023-01-11T21:53:37.0244053Z 
2023-01-11T21:53:37.0244145Z OK
2023-01-11T21:53:37.0244165Z 
2023-01-11T21:53:37.0244269Z Generating XML reports...
2023-01-11T21:53:37.0244720Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214331.xml
2023-01-11T21:53:37.0245201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0245381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0245761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0245951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0245970Z 
2023-01-11T21:53:37.0246079Z Running tests...
2023-01-11T21:53:37.0246338Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0246635Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0246918Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0247669Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.679s)
2023-01-11T21:53:37.0247692Z 
2023-01-11T21:53:37.0247949Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0248058Z Ran 1 test in 1.679s
2023-01-11T21:53:37.0248077Z 
2023-01-11T21:53:37.0248182Z OK (skipped=1)
2023-01-11T21:53:37.0248201Z 
2023-01-11T21:53:37.0248325Z Generating XML reports...
2023-01-11T21:53:37.0248774Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214340.xml
2023-01-11T21:53:37.0249143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0249316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0249684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0249877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0249896Z 
2023-01-11T21:53:37.0250002Z Running tests...
2023-01-11T21:53:37.0250262Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0250573Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0250855Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0251599Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.659s)
2023-01-11T21:53:37.0251622Z 
2023-01-11T21:53:37.0251886Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0251996Z Ran 1 test in 1.660s
2023-01-11T21:53:37.0252015Z 
2023-01-11T21:53:37.0252121Z OK (skipped=1)
2023-01-11T21:53:37.0252140Z 
2023-01-11T21:53:37.0252244Z Generating XML reports...
2023-01-11T21:53:37.0252689Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214344.xml
2023-01-11T21:53:37.0253055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0253230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0253609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0253798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0253817Z 
2023-01-11T21:53:37.0253981Z Running tests...
2023-01-11T21:53:37.0254283Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0254586Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0254867Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0255083Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23559
2023-01-11T21:53:37.0255301Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23560
2023-01-11T21:53:37.0255669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0255842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0256219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0256413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0256778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0256930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0257302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0257487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0257733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0257979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0258377Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0258776Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0259008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0259234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0259553Z STAGE:2023-01-11 21:43:53 23560:23560 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0259885Z STAGE:2023-01-11 21:43:53 23559:23559 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0260124Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0260359Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0260695Z STAGE:2023-01-11 21:43:54 23560:23560 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0261022Z STAGE:2023-01-11 21:43:54 23559:23559 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0261379Z STAGE:2023-01-11 21:43:54 23560:23560 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0261727Z STAGE:2023-01-11 21:43:54 23559:23559 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0262506Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T21:53:37.0263336Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T21:53:37.0263737Z STAGE:2023-01-11 21:43:54 23560:23560 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0264057Z STAGE:2023-01-11 21:43:54 23559:23559 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0264374Z STAGE:2023-01-11 21:43:54 23560:23560 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0264721Z STAGE:2023-01-11 21:43:54 23560:23560 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0265054Z STAGE:2023-01-11 21:43:54 23559:23559 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0265404Z STAGE:2023-01-11 21:43:54 23559:23559 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0265504Z ok (6.071s)
2023-01-11T21:53:37.0265523Z 
2023-01-11T21:53:37.0265782Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0265893Z Ran 1 test in 6.071s
2023-01-11T21:53:37.0265912Z 
2023-01-11T21:53:37.0266002Z OK
2023-01-11T21:53:37.0266021Z 
2023-01-11T21:53:37.0266142Z Generating XML reports...
2023-01-11T21:53:37.0266579Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214348.xml
2023-01-11T21:53:37.0266949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0267123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0267502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0267695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0267714Z 
2023-01-11T21:53:37.0267821Z Running tests...
2023-01-11T21:53:37.0268079Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0268394Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0268648Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0268869Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23681
2023-01-11T21:53:37.0269085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23682
2023-01-11T21:53:37.0269454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0269631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0270007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0270196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0270558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0270729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0271086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0271271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0271518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0271764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0272500Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0272916Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0273145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0273373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0273474Z ok (5.778s)
2023-01-11T21:53:37.0273495Z 
2023-01-11T21:53:37.0273739Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0273852Z Ran 1 test in 5.778s
2023-01-11T21:53:37.0273872Z 
2023-01-11T21:53:37.0273960Z OK
2023-01-11T21:53:37.0273980Z 
2023-01-11T21:53:37.0274101Z Generating XML reports...
2023-01-11T21:53:37.0274555Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214357.xml
2023-01-11T21:53:37.0274936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0275110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0275490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0275660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0275698Z 
2023-01-11T21:53:37.0275786Z Running tests...
2023-01-11T21:53:37.0276045Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0276356Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0276633Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0277383Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.694s)
2023-01-11T21:53:37.0277404Z 
2023-01-11T21:53:37.0277665Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0277777Z Ran 1 test in 1.694s
2023-01-11T21:53:37.0277798Z 
2023-01-11T21:53:37.0277902Z OK (skipped=1)
2023-01-11T21:53:37.0277921Z 
2023-01-11T21:53:37.0278044Z Generating XML reports...
2023-01-11T21:53:37.0278476Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214405.xml
2023-01-11T21:53:37.0278842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0279014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0279400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0279589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0279608Z 
2023-01-11T21:53:37.0279715Z Running tests...
2023-01-11T21:53:37.0279977Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0280290Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0280557Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0280777Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23830
2023-01-11T21:53:37.0280993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23831
2023-01-11T21:53:37.0281358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0281652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0282036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0282225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0282585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0282757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0283112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0283300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0283543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0283795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0284194Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0284587Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0284818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0285044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0285956Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T21:53:37.0286076Z   warnings.warn(
2023-01-11T21:53:37.0286974Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T21:53:37.0287082Z   warnings.warn(
2023-01-11T21:53:37.0287320Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0287551Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0287650Z ok (5.590s)
2023-01-11T21:53:37.0287670Z 
2023-01-11T21:53:37.0287934Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0288043Z Ran 1 test in 5.590s
2023-01-11T21:53:37.0288065Z 
2023-01-11T21:53:37.0288156Z OK
2023-01-11T21:53:37.0288177Z 
2023-01-11T21:53:37.0288301Z Generating XML reports...
2023-01-11T21:53:37.0288735Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214409.xml
2023-01-11T21:53:37.0289100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0289273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0289650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0289841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0289860Z 
2023-01-11T21:53:37.0289966Z Running tests...
2023-01-11T21:53:37.0290226Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0290539Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0290902Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0291656Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.681s)
2023-01-11T21:53:37.0291696Z 
2023-01-11T21:53:37.0291934Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0292044Z Ran 1 test in 1.681s
2023-01-11T21:53:37.0292063Z 
2023-01-11T21:53:37.0292169Z OK (skipped=1)
2023-01-11T21:53:37.0292188Z 
2023-01-11T21:53:37.0292312Z Generating XML reports...
2023-01-11T21:53:37.0292756Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214417.xml
2023-01-11T21:53:37.0293128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0293302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0293678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0293849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0293886Z 
2023-01-11T21:53:37.0293975Z Running tests...
2023-01-11T21:53:37.0294232Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0294544Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0294819Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0295037Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23982
2023-01-11T21:53:37.0295260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23983
2023-01-11T21:53:37.0295627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0295798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0296159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0296348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0296709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0296878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0297248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0297441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0297687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0297929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0298311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0298709Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0298939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0299166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0299502Z STAGE:2023-01-11 21:44:27 23983:23983 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0299924Z STAGE:2023-01-11 21:44:27 23982:23982 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0300171Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0300407Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:53:37.0300745Z STAGE:2023-01-11 21:44:27 23983:23983 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0301064Z STAGE:2023-01-11 21:44:27 23982:23982 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0301411Z STAGE:2023-01-11 21:44:27 23983:23983 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0301757Z STAGE:2023-01-11 21:44:27 23982:23982 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0302084Z STAGE:2023-01-11 21:44:27 23982:23982 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0302420Z STAGE:2023-01-11 21:44:27 23982:23982 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0302766Z STAGE:2023-01-11 21:44:27 23982:23982 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0302867Z ok (6.372s)
2023-01-11T21:53:37.0302887Z 
2023-01-11T21:53:37.0303147Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0303258Z Ran 1 test in 6.372s
2023-01-11T21:53:37.0303278Z 
2023-01-11T21:53:37.0303350Z OK
2023-01-11T21:53:37.0303369Z 
2023-01-11T21:53:37.0303490Z Generating XML reports...
2023-01-11T21:53:37.0303942Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214422.xml
2023-01-11T21:53:37.0304309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0304482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0304865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0305055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0305074Z 
2023-01-11T21:53:37.0305180Z Running tests...
2023-01-11T21:53:37.0305423Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0305736Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0306003Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0306223Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24104
2023-01-11T21:53:37.0306442Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24105
2023-01-11T21:53:37.0306807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0306988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0307365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0307554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0307899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0308069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0308440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0308626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0308872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0309218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0309626Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0310024Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0310254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0310465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0310566Z ok (5.542s)
2023-01-11T21:53:37.0310585Z 
2023-01-11T21:53:37.0310847Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0310957Z Ran 1 test in 5.543s
2023-01-11T21:53:37.0310977Z 
2023-01-11T21:53:37.0311066Z OK
2023-01-11T21:53:37.0311085Z 
2023-01-11T21:53:37.0311209Z Generating XML reports...
2023-01-11T21:53:37.0311663Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214431.xml
2023-01-11T21:53:37.0312269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0312434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0312822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0313008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0313028Z 
2023-01-11T21:53:37.0313136Z Running tests...
2023-01-11T21:53:37.0313403Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0313716Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0313990Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0314217Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24219
2023-01-11T21:53:37.0314417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24220
2023-01-11T21:53:37.0314787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0314957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0315330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0315516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0315880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0316053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0316425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0316613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0316842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0317086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0317481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0317879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0318108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0318334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0318521Z ok (5.652s)
2023-01-11T21:53:37.0318541Z 
2023-01-11T21:53:37.0318856Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0318976Z Ran 1 test in 5.652s
2023-01-11T21:53:37.0318996Z 
2023-01-11T21:53:37.0319068Z OK
2023-01-11T21:53:37.0319087Z 
2023-01-11T21:53:37.0319211Z Generating XML reports...
2023-01-11T21:53:37.0319667Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214439.xml
2023-01-11T21:53:37.0320035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0320210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0320588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0320780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0320802Z 
2023-01-11T21:53:37.0320911Z Running tests...
2023-01-11T21:53:37.0321157Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0321472Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0321751Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0322542Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.637s)
2023-01-11T21:53:37.0322563Z 
2023-01-11T21:53:37.0322823Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0322934Z Ran 1 test in 1.637s
2023-01-11T21:53:37.0322953Z 
2023-01-11T21:53:37.0323061Z OK (skipped=1)
2023-01-11T21:53:37.0323084Z 
2023-01-11T21:53:37.0323206Z Generating XML reports...
2023-01-11T21:53:37.0323655Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214447.xml
2023-01-11T21:53:37.0324025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0324183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0324562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0324749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0324768Z 
2023-01-11T21:53:37.0324873Z Running tests...
2023-01-11T21:53:37.0325132Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0325448Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0325719Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0326463Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.648s)
2023-01-11T21:53:37.0326483Z 
2023-01-11T21:53:37.0326740Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0326832Z Ran 1 test in 1.648s
2023-01-11T21:53:37.0326868Z 
2023-01-11T21:53:37.0326955Z OK (skipped=1)
2023-01-11T21:53:37.0326974Z 
2023-01-11T21:53:37.0327096Z Generating XML reports...
2023-01-11T21:53:37.0327545Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214451.xml
2023-01-11T21:53:37.0327963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0328185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0328563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0328753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0328773Z 
2023-01-11T21:53:37.0328881Z Running tests...
2023-01-11T21:53:37.0329123Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0329436Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0329728Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0330473Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.672s)
2023-01-11T21:53:37.0330496Z 
2023-01-11T21:53:37.0330756Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0330866Z Ran 1 test in 1.672s
2023-01-11T21:53:37.0330884Z 
2023-01-11T21:53:37.0330991Z OK (skipped=1)
2023-01-11T21:53:37.0331009Z 
2023-01-11T21:53:37.0331132Z Generating XML reports...
2023-01-11T21:53:37.0331579Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214455.xml
2023-01-11T21:53:37.0331946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0332103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0332481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0332676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0332695Z 
2023-01-11T21:53:37.0332802Z Running tests...
2023-01-11T21:53:37.0333065Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0333377Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0333676Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0333895Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24436
2023-01-11T21:53:37.0334094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24437
2023-01-11T21:53:37.0334461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0334636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0335021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0335212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0335573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0335746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0336113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0336302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0336531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0336777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0337273Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0337675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0337906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0338135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0338235Z ok (6.151s)
2023-01-11T21:53:37.0338255Z 
2023-01-11T21:53:37.0338516Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0338627Z Ran 1 test in 6.151s
2023-01-11T21:53:37.0338646Z 
2023-01-11T21:53:37.0338719Z OK
2023-01-11T21:53:37.0338739Z 
2023-01-11T21:53:37.0338860Z Generating XML reports...
2023-01-11T21:53:37.0339308Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214459.xml
2023-01-11T21:53:37.0339687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0339860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0340237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0340425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0340443Z 
2023-01-11T21:53:37.0340548Z Running tests...
2023-01-11T21:53:37.0340791Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0341103Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0341376Z   test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0341594Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24555
2023-01-11T21:53:37.0341817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24556
2023-01-11T21:53:37.0342187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0342362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0342736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0342921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0343264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0343440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0343811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0344003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0344248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0344493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0344890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0345285Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0345515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0345874Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2023-01-11T21:53:37.0346127Z   warnings.warn("Initializing zero-element tensors is a no-op")
2023-01-11T21:53:37.0346460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0346840Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2023-01-11T21:53:37.0347090Z   warnings.warn("Initializing zero-element tensors is a no-op")
2023-01-11T21:53:37.0347191Z ok (5.572s)
2023-01-11T21:53:37.0347211Z 
2023-01-11T21:53:37.0347469Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0347580Z Ran 1 test in 5.573s
2023-01-11T21:53:37.0347599Z 
2023-01-11T21:53:37.0347672Z OK
2023-01-11T21:53:37.0347691Z 
2023-01-11T21:53:37.0347812Z Generating XML reports...
2023-01-11T21:53:37.0348261Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214508.xml
2023-01-11T21:53:37.0348629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0348810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0349188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0349377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0349396Z 
2023-01-11T21:53:37.0349503Z Running tests...
2023-01-11T21:53:37.0349745Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0350055Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0350318Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0350538Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24670
2023-01-11T21:53:37.0350756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24671
2023-01-11T21:53:37.0351127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0351298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0351674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0352078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0352442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0352617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0352989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0353177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0353426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0353674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0354073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0354472Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0354699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0354924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:53:37.0355147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0355386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:53:37.0355919Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:37.0356323Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:37.0356426Z ok (4.349s)
2023-01-11T21:53:37.0356446Z 
2023-01-11T21:53:37.0356709Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0356820Z Ran 1 test in 4.349s
2023-01-11T21:53:37.0356839Z 
2023-01-11T21:53:37.0356911Z OK
2023-01-11T21:53:37.0356947Z 
2023-01-11T21:53:37.0357051Z Generating XML reports...
2023-01-11T21:53:37.0357501Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214516.xml
2023-01-11T21:53:37.0357870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0358045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0358430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0358620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0358640Z 
2023-01-11T21:53:37.0358745Z Running tests...
2023-01-11T21:53:37.0359003Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0359301Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0359558Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0359775Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24777
2023-01-11T21:53:37.0359992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24778
2023-01-11T21:53:37.0360359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0360540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0360916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0361104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0361448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0361621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0361989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0362176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0362420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0362674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0363072Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0363467Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0363695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0363920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:53:37.0364143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0364377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:53:37.0364775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:37.0365298Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:37.0365406Z ok (4.393s)
2023-01-11T21:53:37.0365425Z 
2023-01-11T21:53:37.0365689Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0365797Z Ran 1 test in 4.393s
2023-01-11T21:53:37.0365816Z 
2023-01-11T21:53:37.0365907Z OK
2023-01-11T21:53:37.0365926Z 
2023-01-11T21:53:37.0366032Z Generating XML reports...
2023-01-11T21:53:37.0366479Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214523.xml
2023-01-11T21:53:37.0366846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0367020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0367401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0367593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0367612Z 
2023-01-11T21:53:37.0367718Z Running tests...
2023-01-11T21:53:37.0367975Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0368287Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0368550Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0369296Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.659s)
2023-01-11T21:53:37.0369316Z 
2023-01-11T21:53:37.0369573Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0369686Z Ran 1 test in 1.659s
2023-01-11T21:53:37.0369706Z 
2023-01-11T21:53:37.0369811Z OK (skipped=1)
2023-01-11T21:53:37.0369830Z 
2023-01-11T21:53:37.0369950Z Generating XML reports...
2023-01-11T21:53:37.0370399Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214530.xml
2023-01-11T21:53:37.0370764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0370937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0371296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0371488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0371507Z 
2023-01-11T21:53:37.0371613Z Running tests...
2023-01-11T21:53:37.0371872Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0372194Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0372473Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0373216Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78748 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.638s)
2023-01-11T21:53:37.0373236Z 
2023-01-11T21:53:37.0373493Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0373601Z Ran 1 test in 1.638s
2023-01-11T21:53:37.0373620Z 
2023-01-11T21:53:37.0373724Z OK (skipped=1)
2023-01-11T21:53:37.0373743Z 
2023-01-11T21:53:37.0373846Z Generating XML reports...
2023-01-11T21:53:37.0374391Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214534.xml
2023-01-11T21:53:37.0374764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0374938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0375317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0375506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0375526Z 
2023-01-11T21:53:37.0375633Z Running tests...
2023-01-11T21:53:37.0375892Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0376185Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0376459Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0376684Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24952
2023-01-11T21:53:37.0376901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24953
2023-01-11T21:53:37.0377266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0377439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0377814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0378001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0378368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0378520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0378894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0379084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0379330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0379575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0379975Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0380374Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0380602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0380832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0380920Z ok (4.384s)
2023-01-11T21:53:37.0380939Z 
2023-01-11T21:53:37.0381203Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0381313Z Ran 1 test in 4.384s
2023-01-11T21:53:37.0381332Z 
2023-01-11T21:53:37.0381423Z OK
2023-01-11T21:53:37.0381443Z 
2023-01-11T21:53:37.0381566Z Generating XML reports...
2023-01-11T21:53:37.0382011Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214538.xml
2023-01-11T21:53:37.0382378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0382551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0382908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0383098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0383167Z 
2023-01-11T21:53:37.0383283Z Running tests...
2023-01-11T21:53:37.0383580Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0383901Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0384153Z   test_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:37.0384173Z 
2023-01-11T21:53:37.0384429Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0384540Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0384559Z 
2023-01-11T21:53:37.0384664Z OK (skipped=1)
2023-01-11T21:53:37.0384683Z 
2023-01-11T21:53:37.0384787Z Generating XML reports...
2023-01-11T21:53:37.0385235Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214545.xml
2023-01-11T21:53:37.0385604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0385785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0386162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0386353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0386372Z 
2023-01-11T21:53:37.0386479Z Running tests...
2023-01-11T21:53:37.0386735Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0387032Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0387292Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:37.0387311Z 
2023-01-11T21:53:37.0387564Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0387672Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0387692Z 
2023-01-11T21:53:37.0387801Z OK (skipped=1)
2023-01-11T21:53:37.0387820Z 
2023-01-11T21:53:37.0387943Z Generating XML reports...
2023-01-11T21:53:37.0388389Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214547.xml
2023-01-11T21:53:37.0388756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0388928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0389287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0389477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0389496Z 
2023-01-11T21:53:37.0389603Z Running tests...
2023-01-11T21:53:37.0389860Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0390173Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0390431Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0390653Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25123
2023-01-11T21:53:37.0390868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25124
2023-01-11T21:53:37.0391236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0391395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0391771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0392137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0392513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0392815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0393198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0393388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0393635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0393861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0394260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0394656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0394886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0395118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0395456Z STAGE:2023-01-11 21:45:55 25124:25124 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0395780Z STAGE:2023-01-11 21:45:55 25123:25123 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0396115Z STAGE:2023-01-11 21:45:55 25123:25123 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0396463Z STAGE:2023-01-11 21:45:55 25123:25123 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0396778Z STAGE:2023-01-11 21:45:55 25124:25124 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0397124Z STAGE:2023-01-11 21:45:55 25124:25124 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0397450Z STAGE:2023-01-11 21:45:56 25123:25123 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0397779Z STAGE:2023-01-11 21:45:56 25124:25124 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0398115Z STAGE:2023-01-11 21:45:56 25124:25124 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0398461Z STAGE:2023-01-11 21:45:56 25124:25124 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0398791Z STAGE:2023-01-11 21:45:56 25123:25123 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0399132Z STAGE:2023-01-11 21:45:56 25123:25123 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0399232Z ok (6.406s)
2023-01-11T21:53:37.0399253Z 
2023-01-11T21:53:37.0399499Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0399610Z Ran 1 test in 6.406s
2023-01-11T21:53:37.0399629Z 
2023-01-11T21:53:37.0399720Z OK
2023-01-11T21:53:37.0399739Z 
2023-01-11T21:53:37.0399860Z Generating XML reports...
2023-01-11T21:53:37.0400320Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214550.xml
2023-01-11T21:53:37.0400692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0400866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0401241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0401430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0401449Z 
2023-01-11T21:53:37.0401541Z Running tests...
2023-01-11T21:53:37.0401799Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0402111Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0402373Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:37.0402479Z 
2023-01-11T21:53:37.0402745Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0402855Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0402874Z 
2023-01-11T21:53:37.0402980Z OK (skipped=1)
2023-01-11T21:53:37.0402998Z 
2023-01-11T21:53:37.0403120Z Generating XML reports...
2023-01-11T21:53:37.0403550Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214559.xml
2023-01-11T21:53:37.0403919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0404089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0404464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0404654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0404676Z 
2023-01-11T21:53:37.0404785Z Running tests...
2023-01-11T21:53:37.0405046Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0405357Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0405616Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:37.0405635Z 
2023-01-11T21:53:37.0405874Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0405985Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0406003Z 
2023-01-11T21:53:37.0406108Z OK (skipped=1)
2023-01-11T21:53:37.0406127Z 
2023-01-11T21:53:37.0406247Z Generating XML reports...
2023-01-11T21:53:37.0406696Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214601.xml
2023-01-11T21:53:37.0407071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0407248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0407624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0407817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0407836Z 
2023-01-11T21:53:37.0407924Z Running tests...
2023-01-11T21:53:37.0408183Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0408496Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0408753Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0408968Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25312
2023-01-11T21:53:37.0409186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25313
2023-01-11T21:53:37.0409564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0409736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0410097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0410287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0410648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0410820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0411189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0411375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0411725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0411976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0412377Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0412758Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0412987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0413216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0413317Z ok (5.808s)
2023-01-11T21:53:37.0413337Z 
2023-01-11T21:53:37.0413599Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0413707Z Ran 1 test in 5.808s
2023-01-11T21:53:37.0413730Z 
2023-01-11T21:53:37.0413821Z OK
2023-01-11T21:53:37.0413843Z 
2023-01-11T21:53:37.0413964Z Generating XML reports...
2023-01-11T21:53:37.0414397Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214604.xml
2023-01-11T21:53:37.0414763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0414939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0415317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0415506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0415525Z 
2023-01-11T21:53:37.0415632Z Running tests...
2023-01-11T21:53:37.0415888Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0416200Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0416477Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0417203Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.648s)
2023-01-11T21:53:37.0417242Z 
2023-01-11T21:53:37.0417484Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0417594Z Ran 1 test in 1.648s
2023-01-11T21:53:37.0417614Z 
2023-01-11T21:53:37.0417720Z OK (skipped=1)
2023-01-11T21:53:37.0417739Z 
2023-01-11T21:53:37.0417861Z Generating XML reports...
2023-01-11T21:53:37.0418305Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214612.xml
2023-01-11T21:53:37.0418679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0418853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0419230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0419418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0419437Z 
2023-01-11T21:53:37.0419525Z Running tests...
2023-01-11T21:53:37.0419784Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0420098Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0420349Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0420570Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25464
2023-01-11T21:53:37.0420878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25465
2023-01-11T21:53:37.0421251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0421423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0421784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0421979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0422337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0422555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0422928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0423117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0423369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0423612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0424013Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0424393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0424624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0424867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:53:37.0425127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0425371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:53:37.0425775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:37.0426166Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:37.0426267Z ok (4.465s)
2023-01-11T21:53:37.0426287Z 
2023-01-11T21:53:37.0426549Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0426642Z Ran 1 test in 4.466s
2023-01-11T21:53:37.0426661Z 
2023-01-11T21:53:37.0426753Z OK
2023-01-11T21:53:37.0426773Z 
2023-01-11T21:53:37.0426895Z Generating XML reports...
2023-01-11T21:53:37.0427342Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214616.xml
2023-01-11T21:53:37.0427710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0427890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0428268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0428456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0428475Z 
2023-01-11T21:53:37.0428564Z Running tests...
2023-01-11T21:53:37.0428825Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0429137Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0429385Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0429605Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25571
2023-01-11T21:53:37.0429823Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25572
2023-01-11T21:53:37.0430290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0430470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0430849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0431022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0431383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0431554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0432146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0432346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0432598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0432847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0433259Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0433639Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0433869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0434100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0434202Z ok (5.758s)
2023-01-11T21:53:37.0434223Z 
2023-01-11T21:53:37.0434486Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0434596Z Ran 1 test in 5.758s
2023-01-11T21:53:37.0434615Z 
2023-01-11T21:53:37.0434708Z OK
2023-01-11T21:53:37.0434726Z 
2023-01-11T21:53:37.0434852Z Generating XML reports...
2023-01-11T21:53:37.0435301Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214623.xml
2023-01-11T21:53:37.0435653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0435825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0436201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0436388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0436407Z 
2023-01-11T21:53:37.0436517Z Running tests...
2023-01-11T21:53:37.0436775Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0437089Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0437343Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0437545Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25686
2023-01-11T21:53:37.0437765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25687
2023-01-11T21:53:37.0438132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0438305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0438683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0438869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0439230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0439492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0439902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0440101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0440348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0440591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0440991Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0441384Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0441611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0441844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0441945Z ok (4.481s)
2023-01-11T21:53:37.0441966Z 
2023-01-11T21:53:37.0442208Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0442318Z Ran 1 test in 4.481s
2023-01-11T21:53:37.0442337Z 
2023-01-11T21:53:37.0442427Z OK
2023-01-11T21:53:37.0442447Z 
2023-01-11T21:53:37.0442568Z Generating XML reports...
2023-01-11T21:53:37.0443018Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214631.xml
2023-01-11T21:53:37.0443385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0443559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0443936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0444134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0444154Z 
2023-01-11T21:53:37.0444243Z Running tests...
2023-01-11T21:53:37.0444504Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0444816Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0445087Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0445303Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25791
2023-01-11T21:53:37.0445519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25792
2023-01-11T21:53:37.0445882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0446054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0446413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0446608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0446968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0447142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0447510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0447696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0447944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0448188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0448585Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0449061Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0449294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0449531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:53:37.0449759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0449999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:53:37.0450396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:37.0450792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:37.0450895Z ok (4.394s)
2023-01-11T21:53:37.0450918Z 
2023-01-11T21:53:37.0451180Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0451272Z Ran 1 test in 4.394s
2023-01-11T21:53:37.0451291Z 
2023-01-11T21:53:37.0451381Z OK
2023-01-11T21:53:37.0451399Z 
2023-01-11T21:53:37.0451518Z Generating XML reports...
2023-01-11T21:53:37.0451967Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214638.xml
2023-01-11T21:53:37.0452333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0452505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0452882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0453072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0453095Z 
2023-01-11T21:53:37.0453184Z Running tests...
2023-01-11T21:53:37.0453446Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0453758Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0454020Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0454238Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25898
2023-01-11T21:53:37.0454453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25899
2023-01-11T21:53:37.0454815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0454989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0455362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0455540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0455899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0456070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0456435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0456621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0456866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0457109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0457504Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0457976Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0458213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0458454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:53:37.0458676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0458912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:53:37.0459313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:37.0459703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:37.0459803Z ok (4.365s)
2023-01-11T21:53:37.0459823Z 
2023-01-11T21:53:37.0460091Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0460183Z Ran 1 test in 4.365s
2023-01-11T21:53:37.0460220Z 
2023-01-11T21:53:37.0460306Z OK
2023-01-11T21:53:37.0460325Z 
2023-01-11T21:53:37.0460447Z Generating XML reports...
2023-01-11T21:53:37.0460894Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214645.xml
2023-01-11T21:53:37.0461260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0461432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0461809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0461996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0462015Z 
2023-01-11T21:53:37.0462119Z Running tests...
2023-01-11T21:53:37.0462368Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0462680Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0462944Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0463161Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26005
2023-01-11T21:53:37.0463374Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26006
2023-01-11T21:53:37.0463739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0463912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0464290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0464461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0464829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0465000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0465373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0465559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0465802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0466046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0466442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0466831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0467135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0467368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0467470Z ok (5.661s)
2023-01-11T21:53:37.0467490Z 
2023-01-11T21:53:37.0467754Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0467864Z Ran 1 test in 5.661s
2023-01-11T21:53:37.0467882Z 
2023-01-11T21:53:37.0467973Z OK
2023-01-11T21:53:37.0467992Z 
2023-01-11T21:53:37.0468115Z Generating XML reports...
2023-01-11T21:53:37.0468566Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214652.xml
2023-01-11T21:53:37.0468931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0469085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0469466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0469655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0469674Z 
2023-01-11T21:53:37.0469780Z Running tests...
2023-01-11T21:53:37.0470039Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0470353Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0470587Z   test_irecv (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support irecv (0.002s)
2023-01-11T21:53:37.0470606Z 
2023-01-11T21:53:37.0470864Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0470956Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0470993Z 
2023-01-11T21:53:37.0471082Z OK (skipped=1)
2023-01-11T21:53:37.0471101Z 
2023-01-11T21:53:37.0471222Z Generating XML reports...
2023-01-11T21:53:37.0471676Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214700.xml
2023-01-11T21:53:37.0472276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0472460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0472843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0473033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0473053Z 
2023-01-11T21:53:37.0473160Z Running tests...
2023-01-11T21:53:37.0473400Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0473714Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0473952Z   test_isend (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2023-01-11T21:53:37.0473980Z 
2023-01-11T21:53:37.0474241Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0474352Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0474371Z 
2023-01-11T21:53:37.0474475Z OK (skipped=1)
2023-01-11T21:53:37.0474494Z 
2023-01-11T21:53:37.0474614Z Generating XML reports...
2023-01-11T21:53:37.0475061Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214703.xml
2023-01-11T21:53:37.0475428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0475585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0475964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0476152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0476250Z 
2023-01-11T21:53:37.0476422Z Running tests...
2023-01-11T21:53:37.0476700Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0477010Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0477273Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2023-01-11T21:53:37.0477293Z 
2023-01-11T21:53:37.0477548Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0477639Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0477681Z 
2023-01-11T21:53:37.0477769Z OK (skipped=1)
2023-01-11T21:53:37.0477787Z 
2023-01-11T21:53:37.0477910Z Generating XML reports...
2023-01-11T21:53:37.0478356Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214705.xml
2023-01-11T21:53:37.0478720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0478902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0479281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0479470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0479489Z 
2023-01-11T21:53:37.0479596Z Running tests...
2023-01-11T21:53:37.0479836Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0480147Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0480408Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2023-01-11T21:53:37.0480428Z 
2023-01-11T21:53:37.0480683Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0480792Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0480814Z 
2023-01-11T21:53:37.0480924Z OK (skipped=1)
2023-01-11T21:53:37.0480943Z 
2023-01-11T21:53:37.0481068Z Generating XML reports...
2023-01-11T21:53:37.0481515Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214707.xml
2023-01-11T21:53:37.0481883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0482040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0482417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0482602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0482620Z 
2023-01-11T21:53:37.0482726Z Running tests...
2023-01-11T21:53:37.0482983Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0483295Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0483587Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0483804Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26255
2023-01-11T21:53:37.0484004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26256
2023-01-11T21:53:37.0484371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0484541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0484915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0485100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0485462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0485729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0486107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0486292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0486520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0486766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0487165Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0487561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0487794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0488040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:53:37.0488264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0488499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:53:37.0488895Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:37.0489268Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:37.0489505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:53:37.0489745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:53:37.0490149Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:53:37.0490545Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:53:37.0490775Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2023-01-11T21:53:37.0490878Z ok (21.628s)
2023-01-11T21:53:37.0490898Z 
2023-01-11T21:53:37.0491158Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0491270Z Ran 1 test in 21.628s
2023-01-11T21:53:37.0491291Z 
2023-01-11T21:53:37.0491364Z OK
2023-01-11T21:53:37.0491383Z 
2023-01-11T21:53:37.0491505Z Generating XML reports...
2023-01-11T21:53:37.0491951Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214710.xml
2023-01-11T21:53:37.0492321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0492499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0492878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0493064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0493083Z 
2023-01-11T21:53:37.0493189Z Running tests...
2023-01-11T21:53:37.0493432Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0493746Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0494048Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0494269Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26378
2023-01-11T21:53:37.0494485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26379
2023-01-11T21:53:37.0494947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0495126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0495504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0495691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0496034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0496205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0496572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0496756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0497009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0497253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0497649Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0498044Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0498270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0498493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:53:37.0498718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0498957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:53:37.0499361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:37.0499754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:37.0499991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:53:37.0500228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:53:37.0500622Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:53:37.0501011Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:53:37.0501226Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2023-01-11T21:53:37.0501464Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2023-01-11T21:53:37.0501566Z ok (21.484s)
2023-01-11T21:53:37.0501586Z 
2023-01-11T21:53:37.0501847Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0501961Z Ran 1 test in 21.484s
2023-01-11T21:53:37.0501980Z 
2023-01-11T21:53:37.0502070Z OK
2023-01-11T21:53:37.0502089Z 
2023-01-11T21:53:37.0502212Z Generating XML reports...
2023-01-11T21:53:37.0502659Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214734.xml
2023-01-11T21:53:37.0503013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0503185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0503560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0503839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0503860Z 
2023-01-11T21:53:37.0503969Z Running tests...
2023-01-11T21:53:37.0504229Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0504542Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0504951Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:53:37.0504971Z 
2023-01-11T21:53:37.0505230Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0505321Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0505341Z 
2023-01-11T21:53:37.0505451Z OK (skipped=1)
2023-01-11T21:53:37.0505469Z 
2023-01-11T21:53:37.0505593Z Generating XML reports...
2023-01-11T21:53:37.0506041Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214758.xml
2023-01-11T21:53:37.0506419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0506592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0506968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0507157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0507176Z 
2023-01-11T21:53:37.0507283Z Running tests...
2023-01-11T21:53:37.0507525Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0507832Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0508219Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:53:37.0508237Z 
2023-01-11T21:53:37.0508500Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0508615Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0508635Z 
2023-01-11T21:53:37.0508741Z OK (skipped=1)
2023-01-11T21:53:37.0508768Z 
2023-01-11T21:53:37.0508989Z Generating XML reports...
2023-01-11T21:53:37.0509846Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214800.xml
2023-01-11T21:53:37.0510318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0510476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0510854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0511044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0511065Z 
2023-01-11T21:53:37.0511172Z Running tests...
2023-01-11T21:53:37.0511446Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0511757Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0512373Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:53:37.0512395Z 
2023-01-11T21:53:37.0512654Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0512748Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0512792Z 
2023-01-11T21:53:37.0512881Z OK (skipped=1)
2023-01-11T21:53:37.0512900Z 
2023-01-11T21:53:37.0513023Z Generating XML reports...
2023-01-11T21:53:37.0513471Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214803.xml
2023-01-11T21:53:37.0513838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0514206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0514598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0514786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0514807Z 
2023-01-11T21:53:37.0514915Z Running tests...
2023-01-11T21:53:37.0515157Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0515468Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0515879Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:53:37.0515899Z 
2023-01-11T21:53:37.0516155Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0516265Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0516287Z 
2023-01-11T21:53:37.0516397Z OK (skipped=1)
2023-01-11T21:53:37.0516415Z 
2023-01-11T21:53:37.0516542Z Generating XML reports...
2023-01-11T21:53:37.0516987Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214805.xml
2023-01-11T21:53:37.0517352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0517508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0517882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0518070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0518089Z 
2023-01-11T21:53:37.0518195Z Running tests...
2023-01-11T21:53:37.0518454Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0518765Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0519177Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:53:37.0519197Z 
2023-01-11T21:53:37.0519529Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0519638Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0519657Z 
2023-01-11T21:53:37.0519744Z OK (skipped=1)
2023-01-11T21:53:37.0519763Z 
2023-01-11T21:53:37.0519887Z Generating XML reports...
2023-01-11T21:53:37.0520333Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214807.xml
2023-01-11T21:53:37.0520702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0520875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0521254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0521454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0521473Z 
2023-01-11T21:53:37.0521581Z Running tests...
2023-01-11T21:53:37.0521824Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0522136Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0522415Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0522683Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26666
2023-01-11T21:53:37.0522903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26667
2023-01-11T21:53:37.0523270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0523444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0523920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0524119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0524465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0524641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0525062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0525275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0525674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0525957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0526409Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0526843Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0527110Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0527321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0527455Z ok (5.161s)
2023-01-11T21:53:37.0527476Z 
2023-01-11T21:53:37.0527784Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0527929Z Ran 1 test in 5.161s
2023-01-11T21:53:37.0527948Z 
2023-01-11T21:53:37.0528111Z OK
2023-01-11T21:53:37.0528131Z 
2023-01-11T21:53:37.0528288Z Generating XML reports...
2023-01-11T21:53:37.0528775Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214810.xml
2023-01-11T21:53:37.0529225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0529435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0529797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0530031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0530050Z 
2023-01-11T21:53:37.0530191Z Running tests...
2023-01-11T21:53:37.0530488Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0530869Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0531184Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0531437Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26780
2023-01-11T21:53:37.0531696Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26781
2023-01-11T21:53:37.0532055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0532273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0532689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0532913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0533309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0533549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0533958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0534319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0534555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0534835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0535271Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0535701Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0535963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0536222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0536391Z ok (5.191s)
2023-01-11T21:53:37.0536411Z 
2023-01-11T21:53:37.0536731Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0536877Z Ran 1 test in 5.191s
2023-01-11T21:53:37.0536897Z 
2023-01-11T21:53:37.0536971Z OK
2023-01-11T21:53:37.0537042Z 
2023-01-11T21:53:37.0537149Z Generating XML reports...
2023-01-11T21:53:37.0537634Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214818.xml
2023-01-11T21:53:37.0538035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0538244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0538654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0538920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0538940Z 
2023-01-11T21:53:37.0539118Z Running tests...
2023-01-11T21:53:37.0539421Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0539720Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0540028Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0540279Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26894
2023-01-11T21:53:37.0540529Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26895
2023-01-11T21:53:37.0540933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0541149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0541588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0541812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0542166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0542375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0542776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0542996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0543276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0543566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0544002Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0544467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0544862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0545078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0545213Z ok (5.697s)
2023-01-11T21:53:37.0545233Z 
2023-01-11T21:53:37.0545534Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0545689Z Ran 1 test in 5.697s
2023-01-11T21:53:37.0545709Z 
2023-01-11T21:53:37.0545838Z OK
2023-01-11T21:53:37.0545858Z 
2023-01-11T21:53:37.0546016Z Generating XML reports...
2023-01-11T21:53:37.0546501Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214825.xml
2023-01-11T21:53:37.0546933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0547145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0547516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0547743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0547763Z 
2023-01-11T21:53:37.0547913Z Running tests...
2023-01-11T21:53:37.0548208Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0548554Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0548862Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0549139Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27009
2023-01-11T21:53:37.0549422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27010
2023-01-11T21:53:37.0549780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0549997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0550419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0550642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0551034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0551239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0551644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0552062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0552310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0552648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0553104Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0553538Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0553800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0554062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0554196Z ok (5.250s)
2023-01-11T21:53:37.0554217Z 
2023-01-11T21:53:37.0554517Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0554692Z Ran 1 test in 5.250s
2023-01-11T21:53:37.0554713Z 
2023-01-11T21:53:37.0554786Z OK
2023-01-11T21:53:37.0554804Z 
2023-01-11T21:53:37.0555005Z Generating XML reports...
2023-01-11T21:53:37.0555643Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214833.xml
2023-01-11T21:53:37.0556055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0556263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0556677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0556900Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0556920Z 
2023-01-11T21:53:37.0557069Z Running tests...
2023-01-11T21:53:37.0557362Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0557658Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0558008Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0558805Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81022 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.649s)
2023-01-11T21:53:37.0558827Z 
2023-01-11T21:53:37.0559124Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0559270Z Ran 1 test in 1.649s
2023-01-11T21:53:37.0559290Z 
2023-01-11T21:53:37.0559431Z OK (skipped=1)
2023-01-11T21:53:37.0559450Z 
2023-01-11T21:53:37.0559645Z Generating XML reports...
2023-01-11T21:53:37.0560131Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214841.xml
2023-01-11T21:53:37.0560538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0560703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0561159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0561381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0561401Z 
2023-01-11T21:53:37.0561539Z Running tests...
2023-01-11T21:53:37.0561832Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0562186Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0562474Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T21:53:37.0562494Z 
2023-01-11T21:53:37.0562806Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0562952Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0562971Z 
2023-01-11T21:53:37.0563063Z OK (skipped=1)
2023-01-11T21:53:37.0563082Z 
2023-01-11T21:53:37.0563276Z Generating XML reports...
2023-01-11T21:53:37.0563765Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214845.xml
2023-01-11T21:53:37.0564168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0564386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0564799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0565055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0565075Z 
2023-01-11T21:53:37.0565215Z Running tests...
2023-01-11T21:53:37.0565510Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0565806Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0566254Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T21:53:37.0566275Z 
2023-01-11T21:53:37.0566583Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0566729Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0566749Z 
2023-01-11T21:53:37.0566892Z OK (skipped=1)
2023-01-11T21:53:37.0566911Z 
2023-01-11T21:53:37.0567067Z Generating XML reports...
2023-01-11T21:53:37.0567557Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214848.xml
2023-01-11T21:53:37.0577951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0578185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0578597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0578804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0578827Z 
2023-01-11T21:53:37.0578935Z Running tests...
2023-01-11T21:53:37.0579202Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0579517Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0579826Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T21:53:37.0579846Z 
2023-01-11T21:53:37.0580105Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0580215Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0580235Z 
2023-01-11T21:53:37.0580340Z OK (skipped=1)
2023-01-11T21:53:37.0580360Z 
2023-01-11T21:53:37.0580465Z Generating XML reports...
2023-01-11T21:53:37.0580915Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214850.xml
2023-01-11T21:53:37.0581289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0581464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0581841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0582031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0582050Z 
2023-01-11T21:53:37.0582156Z Running tests...
2023-01-11T21:53:37.0582415Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0582709Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0583010Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0583232Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27256
2023-01-11T21:53:37.0583449Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27257
2023-01-11T21:53:37.0583814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0583986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0584360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0584548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0584905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0585058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0585527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0585765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0586010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0586254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0586653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0587048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0587276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0587501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0587585Z ok (4.309s)
2023-01-11T21:53:37.0587608Z 
2023-01-11T21:53:37.0587874Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0587983Z Ran 1 test in 4.309s
2023-01-11T21:53:37.0588003Z 
2023-01-11T21:53:37.0588094Z OK
2023-01-11T21:53:37.0588113Z 
2023-01-11T21:53:37.0588234Z Generating XML reports...
2023-01-11T21:53:37.0588679Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214853.xml
2023-01-11T21:53:37.0589044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0589216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0589573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0589762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0589781Z 
2023-01-11T21:53:37.0589890Z Running tests...
2023-01-11T21:53:37.0590153Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0590462Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0590753Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0590971Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27361
2023-01-11T21:53:37.0591186Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27362
2023-01-11T21:53:37.0591555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0591709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0592422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0592625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0592994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0593164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0593533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0593723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0593965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0594191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0594587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0595079Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0595375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0595600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0595702Z ok (4.258s)
2023-01-11T21:53:37.0595723Z 
2023-01-11T21:53:37.0595987Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0596097Z Ran 1 test in 4.258s
2023-01-11T21:53:37.0596117Z 
2023-01-11T21:53:37.0596207Z OK
2023-01-11T21:53:37.0596226Z 
2023-01-11T21:53:37.0596330Z Generating XML reports...
2023-01-11T21:53:37.0596780Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214859.xml
2023-01-11T21:53:37.0597146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0597331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0598184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0598375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0598395Z 
2023-01-11T21:53:37.0598504Z Running tests...
2023-01-11T21:53:37.0598761Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0599068Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0599325Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T21:53:37.0599344Z 
2023-01-11T21:53:37.0599599Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0599708Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0599727Z 
2023-01-11T21:53:37.0599836Z OK (skipped=1)
2023-01-11T21:53:37.0599856Z 
2023-01-11T21:53:37.0599979Z Generating XML reports...
2023-01-11T21:53:37.0600422Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214906.xml
2023-01-11T21:53:37.0600791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0600965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0601340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0601510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0601529Z 
2023-01-11T21:53:37.0601635Z Running tests...
2023-01-11T21:53:37.0601891Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0602198Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0602500Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T21:53:37.0602521Z 
2023-01-11T21:53:37.0602776Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0602884Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0602903Z 
2023-01-11T21:53:37.0603005Z OK (skipped=1)
2023-01-11T21:53:37.0603024Z 
2023-01-11T21:53:37.0603144Z Generating XML reports...
2023-01-11T21:53:37.0603569Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214909.xml
2023-01-11T21:53:37.0603930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0604104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0604478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0604785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0604806Z 
2023-01-11T21:53:37.0604917Z Running tests...
2023-01-11T21:53:37.0605177Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0605485Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0605741Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0606493Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.677s)
2023-01-11T21:53:37.0606532Z 
2023-01-11T21:53:37.0606772Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0606891Z Ran 1 test in 1.678s
2023-01-11T21:53:37.0606910Z 
2023-01-11T21:53:37.0607015Z OK (skipped=1)
2023-01-11T21:53:37.0607034Z 
2023-01-11T21:53:37.0607153Z Generating XML reports...
2023-01-11T21:53:37.0607599Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214911.xml
2023-01-11T21:53:37.0607963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0608132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0608506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0608679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0608698Z 
2023-01-11T21:53:37.0608805Z Running tests...
2023-01-11T21:53:37.0609060Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0609375Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0609652Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0609868Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27566
2023-01-11T21:53:37.0610083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27567
2023-01-11T21:53:37.0610445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0610600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0610975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0611162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0611525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0611693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0612065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0612253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0612492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0612736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0613112Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0613504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0613828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0614059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0614162Z ok (6.226s)
2023-01-11T21:53:37.0614182Z 
2023-01-11T21:53:37.0614446Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0614556Z Ran 1 test in 6.226s
2023-01-11T21:53:37.0614575Z 
2023-01-11T21:53:37.0614664Z OK
2023-01-11T21:53:37.0614683Z 
2023-01-11T21:53:37.0614805Z Generating XML reports...
2023-01-11T21:53:37.0615234Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214915.xml
2023-01-11T21:53:37.0615601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0615772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0616153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0616343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0616363Z 
2023-01-11T21:53:37.0616468Z Running tests...
2023-01-11T21:53:37.0616726Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0617038Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0617288Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0617505Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27685
2023-01-11T21:53:37.0617719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27686
2023-01-11T21:53:37.0618086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0618262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0618631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0618819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0619176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0619348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0619697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0619882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0620123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0620366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0620762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0621154Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0621380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0621603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0621685Z ok (5.681s)
2023-01-11T21:53:37.0621722Z 
2023-01-11T21:53:37.0621964Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0622072Z Ran 1 test in 5.681s
2023-01-11T21:53:37.0622091Z 
2023-01-11T21:53:37.0622180Z OK
2023-01-11T21:53:37.0622199Z 
2023-01-11T21:53:37.0622318Z Generating XML reports...
2023-01-11T21:53:37.0622908Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214924.xml
2023-01-11T21:53:37.0623285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0623458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0623835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0624003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0624023Z 
2023-01-11T21:53:37.0624130Z Running tests...
2023-01-11T21:53:37.0624389Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0624698Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0624983Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0625208Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27800
2023-01-11T21:53:37.0625423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27801
2023-01-11T21:53:37.0625787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0625942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0626315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0626496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0626854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0627023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0627393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0627579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0627819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0628059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0628436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0628828Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0629054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0629277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0629380Z ok (5.695s)
2023-01-11T21:53:37.0629400Z 
2023-01-11T21:53:37.0629663Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0629775Z Ran 1 test in 5.695s
2023-01-11T21:53:37.0629795Z 
2023-01-11T21:53:37.0629887Z OK
2023-01-11T21:53:37.0629906Z 
2023-01-11T21:53:37.0630028Z Generating XML reports...
2023-01-11T21:53:37.0630454Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214932.xml
2023-01-11T21:53:37.0630821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0630993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0631369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0631559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0631625Z 
2023-01-11T21:53:37.0631736Z Running tests...
2023-01-11T21:53:37.0632307Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0632644Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0632905Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0633654Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.657s)
2023-01-11T21:53:37.0633693Z 
2023-01-11T21:53:37.0633931Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0634040Z Ran 1 test in 1.657s
2023-01-11T21:53:37.0634060Z 
2023-01-11T21:53:37.0634162Z OK (skipped=1)
2023-01-11T21:53:37.0634184Z 
2023-01-11T21:53:37.0634305Z Generating XML reports...
2023-01-11T21:53:37.0634752Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214941.xml
2023-01-11T21:53:37.0635116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0635289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0635661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0635830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0635848Z 
2023-01-11T21:53:37.0635958Z Running tests...
2023-01-11T21:53:37.0636215Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0636523Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0636820Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0637562Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.658s)
2023-01-11T21:53:37.0637582Z 
2023-01-11T21:53:37.0637839Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0637945Z Ran 1 test in 1.659s
2023-01-11T21:53:37.0637964Z 
2023-01-11T21:53:37.0638069Z OK (skipped=1)
2023-01-11T21:53:37.0638088Z 
2023-01-11T21:53:37.0638207Z Generating XML reports...
2023-01-11T21:53:37.0638631Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214945.xml
2023-01-11T21:53:37.0639000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0639177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0639553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0639741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0639760Z 
2023-01-11T21:53:37.0639864Z Running tests...
2023-01-11T21:53:37.0640122Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0640429Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0640717Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0640934Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27983
2023-01-11T21:53:37.0641260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27984
2023-01-11T21:53:37.0641635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0641807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0642176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0642365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0642720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0642891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0643238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0643422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0643671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0643912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0644311Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0644703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0644928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0645154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0645284Z skip: Need at least 4 CUDA devices (4.294s)
2023-01-11T21:53:37.0645321Z 
2023-01-11T21:53:37.0645565Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0645680Z Ran 1 test in 4.294s
2023-01-11T21:53:37.0645699Z 
2023-01-11T21:53:37.0645806Z OK (skipped=1)
2023-01-11T21:53:37.0645825Z 
2023-01-11T21:53:37.0645944Z Generating XML reports...
2023-01-11T21:53:37.0646388Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214949.xml
2023-01-11T21:53:37.0646754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0646925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0647295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0647463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0647500Z 
2023-01-11T21:53:37.0647588Z Running tests...
2023-01-11T21:53:37.0647847Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0648161Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0648486Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0648699Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28088
2023-01-11T21:53:37.0648910Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28089
2023-01-11T21:53:37.0649271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0649443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0649799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0649988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0650448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0650625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0650995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0651182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0651424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0651664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0652039Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0652432Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0652666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0652893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0653039Z skip: Need at least 4 CUDA devices (4.267s)
2023-01-11T21:53:37.0653060Z 
2023-01-11T21:53:37.0653320Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0653429Z Ran 1 test in 4.267s
2023-01-11T21:53:37.0653448Z 
2023-01-11T21:53:37.0653552Z OK (skipped=1)
2023-01-11T21:53:37.0653571Z 
2023-01-11T21:53:37.0653691Z Generating XML reports...
2023-01-11T21:53:37.0654123Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214956.xml
2023-01-11T21:53:37.0654488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0654667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0655044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0655231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0655251Z 
2023-01-11T21:53:37.0655355Z Running tests...
2023-01-11T21:53:37.0655614Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0655923Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0656212Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0656938Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.693s)
2023-01-11T21:53:37.0656980Z 
2023-01-11T21:53:37.0657222Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0657332Z Ran 1 test in 1.693s
2023-01-11T21:53:37.0657352Z 
2023-01-11T21:53:37.0657455Z OK (skipped=1)
2023-01-11T21:53:37.0657473Z 
2023-01-11T21:53:37.0657594Z Generating XML reports...
2023-01-11T21:53:37.0658040Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215003.xml
2023-01-11T21:53:37.0658405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0658578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0658952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0659123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0659210Z 
2023-01-11T21:53:37.0659338Z Running tests...
2023-01-11T21:53:37.0659607Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0659919Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0660190Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:37.0660209Z 
2023-01-11T21:53:37.0660468Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0660577Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0660595Z 
2023-01-11T21:53:37.0660698Z OK (skipped=1)
2023-01-11T21:53:37.0660717Z 
2023-01-11T21:53:37.0660838Z Generating XML reports...
2023-01-11T21:53:37.0661265Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215007.xml
2023-01-11T21:53:37.0661635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0661812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0662183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0662368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0662387Z 
2023-01-11T21:53:37.0662492Z Running tests...
2023-01-11T21:53:37.0662752Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0663061Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0663329Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:37.0663348Z 
2023-01-11T21:53:37.0663588Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0663701Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0663720Z 
2023-01-11T21:53:37.0663825Z OK (skipped=1)
2023-01-11T21:53:37.0663845Z 
2023-01-11T21:53:37.0663964Z Generating XML reports...
2023-01-11T21:53:37.0664405Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215009.xml
2023-01-11T21:53:37.0664769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0664939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0665308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0665496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0665516Z 
2023-01-11T21:53:37.0665606Z Running tests...
2023-01-11T21:53:37.0665866Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0666183Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0666456Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:37.0666476Z 
2023-01-11T21:53:37.0666733Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0666843Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0666862Z 
2023-01-11T21:53:37.0666967Z OK (skipped=1)
2023-01-11T21:53:37.0666985Z 
2023-01-11T21:53:37.0667108Z Generating XML reports...
2023-01-11T21:53:37.0667531Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215012.xml
2023-01-11T21:53:37.0667896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0668070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0668551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0668747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0668766Z 
2023-01-11T21:53:37.0668873Z Running tests...
2023-01-11T21:53:37.0669129Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0669438Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0669706Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:37.0669725Z 
2023-01-11T21:53:37.0669964Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0670074Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0670093Z 
2023-01-11T21:53:37.0670200Z OK (skipped=1)
2023-01-11T21:53:37.0670219Z 
2023-01-11T21:53:37.0670339Z Generating XML reports...
2023-01-11T21:53:37.0670786Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215014.xml
2023-01-11T21:53:37.0671152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0671326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0671697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0672073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0672095Z 
2023-01-11T21:53:37.0672194Z Running tests...
2023-01-11T21:53:37.0672466Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0672778Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0673042Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:37.0673069Z 
2023-01-11T21:53:37.0673325Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0673433Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0673453Z 
2023-01-11T21:53:37.0673557Z OK (skipped=1)
2023-01-11T21:53:37.0673576Z 
2023-01-11T21:53:37.0673697Z Generating XML reports...
2023-01-11T21:53:37.0674144Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215016.xml
2023-01-11T21:53:37.0674501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0674676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0675052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0675239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0675261Z 
2023-01-11T21:53:37.0675371Z Running tests...
2023-01-11T21:53:37.0675631Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0675943Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0676208Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:37.0676227Z 
2023-01-11T21:53:37.0676463Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0676572Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0676592Z 
2023-01-11T21:53:37.0676696Z OK (skipped=1)
2023-01-11T21:53:37.0676715Z 
2023-01-11T21:53:37.0676835Z Generating XML reports...
2023-01-11T21:53:37.0677281Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215019.xml
2023-01-11T21:53:37.0677647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0677960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0678349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0678539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0678559Z 
2023-01-11T21:53:37.0678648Z Running tests...
2023-01-11T21:53:37.0678904Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0679215Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0679482Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:37.0679501Z 
2023-01-11T21:53:37.0679758Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0679865Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0679888Z 
2023-01-11T21:53:37.0679996Z OK (skipped=1)
2023-01-11T21:53:37.0680015Z 
2023-01-11T21:53:37.0680136Z Generating XML reports...
2023-01-11T21:53:37.0680581Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215021.xml
2023-01-11T21:53:37.0680929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0681102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0681475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0681664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0681684Z 
2023-01-11T21:53:37.0681789Z Running tests...
2023-01-11T21:53:37.0682046Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0682361Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0682627Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:37.0682646Z 
2023-01-11T21:53:37.0682901Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0682993Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0683012Z 
2023-01-11T21:53:37.0683118Z OK (skipped=1)
2023-01-11T21:53:37.0683137Z 
2023-01-11T21:53:37.0683259Z Generating XML reports...
2023-01-11T21:53:37.0683705Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215024.xml
2023-01-11T21:53:37.0684071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0684243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0684619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0684811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0684831Z 
2023-01-11T21:53:37.0684920Z Running tests...
2023-01-11T21:53:37.0685179Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0685490Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0685746Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:37.0685767Z 
2023-01-11T21:53:37.0686024Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0686132Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0686150Z 
2023-01-11T21:53:37.0686255Z OK (skipped=1)
2023-01-11T21:53:37.0686273Z 
2023-01-11T21:53:37.0686392Z Generating XML reports...
2023-01-11T21:53:37.0686878Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215026.xml
2023-01-11T21:53:37.0687273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0687448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0687823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0688010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0688030Z 
2023-01-11T21:53:37.0688137Z Running tests...
2023-01-11T21:53:37.0688397Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0688710Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0688966Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:37.0688989Z 
2023-01-11T21:53:37.0689248Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0689341Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0689360Z 
2023-01-11T21:53:37.0689466Z OK (skipped=1)
2023-01-11T21:53:37.0689484Z 
2023-01-11T21:53:37.0689604Z Generating XML reports...
2023-01-11T21:53:37.0690049Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215028.xml
2023-01-11T21:53:37.0690416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0690587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0690962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0691149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0691168Z 
2023-01-11T21:53:37.0691260Z Running tests...
2023-01-11T21:53:37.0691521Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0691832Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0692092Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0692310Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28557
2023-01-11T21:53:37.0692526Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28558
2023-01-11T21:53:37.0692892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0693063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0693434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0693605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0693977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0694147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0694516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0694702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0694949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0695193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0695589Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0696015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0696295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0696528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0696867Z STAGE:2023-01-11 21:50:36 28557:28557 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0697630Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1777: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2023-01-11T21:53:37.0697743Z   warnings.warn(
2023-01-11T21:53:37.0698076Z STAGE:2023-01-11 21:50:36 28558:28558 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0698841Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1777: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2023-01-11T21:53:37.0698953Z   warnings.warn(
2023-01-11T21:53:37.0699294Z STAGE:2023-01-11 21:50:36 28558:28558 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0699616Z STAGE:2023-01-11 21:50:36 28557:28557 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0699965Z STAGE:2023-01-11 21:50:36 28558:28558 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0700307Z STAGE:2023-01-11 21:50:36 28557:28557 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0700634Z STAGE:2023-01-11 21:50:36 28557:28557 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0700961Z STAGE:2023-01-11 21:50:36 28557:28557 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0701313Z STAGE:2023-01-11 21:50:36 28557:28557 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0701639Z STAGE:2023-01-11 21:50:36 28558:28558 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0701968Z STAGE:2023-01-11 21:50:36 28558:28558 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0702312Z STAGE:2023-01-11 21:50:36 28558:28558 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0702394Z ok (5.713s)
2023-01-11T21:53:37.0702415Z 
2023-01-11T21:53:37.0702678Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0702788Z Ran 1 test in 5.713s
2023-01-11T21:53:37.0702807Z 
2023-01-11T21:53:37.0702896Z OK
2023-01-11T21:53:37.0702915Z 
2023-01-11T21:53:37.0703035Z Generating XML reports...
2023-01-11T21:53:37.0703489Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215031.xml
2023-01-11T21:53:37.0703861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0704035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0704415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0704588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0704607Z 
2023-01-11T21:53:37.0704714Z Running tests...
2023-01-11T21:53:37.0704973Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0705285Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0705546Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:37.0705613Z 
2023-01-11T21:53:37.0705916Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0706034Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0706053Z 
2023-01-11T21:53:37.0706162Z OK (skipped=1)
2023-01-11T21:53:37.0706181Z 
2023-01-11T21:53:37.0706284Z Generating XML reports...
2023-01-11T21:53:37.0706733Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215039.xml
2023-01-11T21:53:37.0707099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0707272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0707645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0707833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0707852Z 
2023-01-11T21:53:37.0707962Z Running tests...
2023-01-11T21:53:37.0708224Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0708536Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0708794Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0709014Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28708
2023-01-11T21:53:37.0709228Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28709
2023-01-11T21:53:37.0709593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0709765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0710139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0710329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0710691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0710844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0711214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0711400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0711646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0712074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0712495Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0712898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0713136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0713364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0713681Z STAGE:2023-01-11 21:50:46 28709:28709 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0714011Z STAGE:2023-01-11 21:50:46 28708:28708 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0714344Z STAGE:2023-01-11 21:50:47 28709:28709 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0714678Z STAGE:2023-01-11 21:50:47 28708:28708 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0715025Z STAGE:2023-01-11 21:50:47 28709:28709 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0715439Z STAGE:2023-01-11 21:50:47 28708:28708 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0715829Z STAGE:2023-01-11 21:50:47 28708:28708 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0716159Z STAGE:2023-01-11 21:50:47 28708:28708 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0716507Z STAGE:2023-01-11 21:50:47 28708:28708 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0716815Z STAGE:2023-01-11 21:50:47 28709:28709 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0717148Z STAGE:2023-01-11 21:50:47 28709:28709 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0717494Z STAGE:2023-01-11 21:50:47 28709:28709 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0717594Z ok (6.471s)
2023-01-11T21:53:37.0717615Z 
2023-01-11T21:53:37.0717879Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0717994Z Ran 1 test in 6.471s
2023-01-11T21:53:37.0718013Z 
2023-01-11T21:53:37.0718102Z OK
2023-01-11T21:53:37.0718121Z 
2023-01-11T21:53:37.0718242Z Generating XML reports...
2023-01-11T21:53:37.0718691Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215041.xml
2023-01-11T21:53:37.0719043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0719216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0719594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0719783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0719802Z 
2023-01-11T21:53:37.0719908Z Running tests...
2023-01-11T21:53:37.0720165Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0720484Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0720755Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0720956Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28827
2023-01-11T21:53:37.0721178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28828
2023-01-11T21:53:37.0721543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0721716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0722088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0722275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0722689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0722864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0723236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0723405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0723654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0723897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0724295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0724690Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0725015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0725297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0725401Z ok (5.754s)
2023-01-11T21:53:37.0725421Z 
2023-01-11T21:53:37.0725665Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0725776Z Ran 1 test in 5.755s
2023-01-11T21:53:37.0725795Z 
2023-01-11T21:53:37.0725888Z OK
2023-01-11T21:53:37.0725907Z 
2023-01-11T21:53:37.0726028Z Generating XML reports...
2023-01-11T21:53:37.0726479Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215050.xml
2023-01-11T21:53:37.0726846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0727019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0727401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0727590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0727609Z 
2023-01-11T21:53:37.0727698Z Running tests...
2023-01-11T21:53:37.0727959Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0728271Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0728526Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:37.0728546Z 
2023-01-11T21:53:37.0728801Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0728911Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0728930Z 
2023-01-11T21:53:37.0729035Z OK (skipped=1)
2023-01-11T21:53:37.0729053Z 
2023-01-11T21:53:37.0729174Z Generating XML reports...
2023-01-11T21:53:37.0729626Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215059.xml
2023-01-11T21:53:37.0729978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0730150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0730526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0730714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0730733Z 
2023-01-11T21:53:37.0730838Z Running tests...
2023-01-11T21:53:37.0731098Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0731409Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0731665Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0731873Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28975
2023-01-11T21:53:37.0732094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28976
2023-01-11T21:53:37.0732457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0732627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0733002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0733190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0733551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0733721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0734140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0734367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0734617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0734861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0735262Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0735651Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0735880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0736108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0736452Z STAGE:2023-01-11 21:51:06 28975:28975 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0736778Z STAGE:2023-01-11 21:51:06 28976:28976 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0737100Z STAGE:2023-01-11 21:51:06 28975:28975 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0737431Z STAGE:2023-01-11 21:51:06 28976:28976 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0737777Z STAGE:2023-01-11 21:51:06 28975:28975 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0738121Z STAGE:2023-01-11 21:51:06 28976:28976 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0738446Z STAGE:2023-01-11 21:51:06 28976:28976 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0738777Z STAGE:2023-01-11 21:51:06 28976:28976 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0739127Z STAGE:2023-01-11 21:51:06 28976:28976 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0739450Z STAGE:2023-01-11 21:51:06 28975:28975 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0739780Z STAGE:2023-01-11 21:51:06 28975:28975 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0740106Z STAGE:2023-01-11 21:51:06 28975:28975 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0740206Z ok (5.765s)
2023-01-11T21:53:37.0740226Z 
2023-01-11T21:53:37.0740486Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0740598Z Ran 1 test in 5.765s
2023-01-11T21:53:37.0740617Z 
2023-01-11T21:53:37.0740707Z OK
2023-01-11T21:53:37.0740726Z 
2023-01-11T21:53:37.0740846Z Generating XML reports...
2023-01-11T21:53:37.0741292Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215101.xml
2023-01-11T21:53:37.0741761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0742053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0742593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0742784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0742805Z 
2023-01-11T21:53:37.0742912Z Running tests...
2023-01-11T21:53:37.0743173Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0743485Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0743752Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0743970Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29093
2023-01-11T21:53:37.0744286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29094
2023-01-11T21:53:37.0744665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0744840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0745213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0745405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0745766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0745936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0746301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0746493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0746724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0746971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0747368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0747762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0747992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0748219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0748551Z STAGE:2023-01-11 21:51:14 29093:29093 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0748880Z STAGE:2023-01-11 21:51:14 29094:29094 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0749213Z STAGE:2023-01-11 21:51:14 29094:29094 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0749548Z STAGE:2023-01-11 21:51:14 29094:29094 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0749881Z STAGE:2023-01-11 21:51:14 29093:29093 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0750225Z STAGE:2023-01-11 21:51:14 29093:29093 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0750550Z STAGE:2023-01-11 21:51:14 29093:29093 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0750879Z STAGE:2023-01-11 21:51:14 29093:29093 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0751223Z STAGE:2023-01-11 21:51:14 29093:29093 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0751553Z STAGE:2023-01-11 21:51:15 29094:29094 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0752073Z STAGE:2023-01-11 21:51:15 29094:29094 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0752436Z STAGE:2023-01-11 21:51:15 29094:29094 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0752519Z ok (5.676s)
2023-01-11T21:53:37.0752540Z 
2023-01-11T21:53:37.0752806Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0752918Z Ran 1 test in 5.676s
2023-01-11T21:53:37.0752937Z 
2023-01-11T21:53:37.0753027Z OK
2023-01-11T21:53:37.0753046Z 
2023-01-11T21:53:37.0753167Z Generating XML reports...
2023-01-11T21:53:37.0753615Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215109.xml
2023-01-11T21:53:37.0754063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0754303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0754668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0754861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0754881Z 
2023-01-11T21:53:37.0754989Z Running tests...
2023-01-11T21:53:37.0755249Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0755563Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0755826Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:37.0755845Z 
2023-01-11T21:53:37.0756103Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0756211Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0756233Z 
2023-01-11T21:53:37.0756341Z OK (skipped=1)
2023-01-11T21:53:37.0756360Z 
2023-01-11T21:53:37.0756464Z Generating XML reports...
2023-01-11T21:53:37.0756915Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215118.xml
2023-01-11T21:53:37.0757282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0757457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0757834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0758023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0758043Z 
2023-01-11T21:53:37.0758147Z Running tests...
2023-01-11T21:53:37.0758402Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0758724Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0758959Z   test_scatter (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:37.0758978Z 
2023-01-11T21:53:37.0759234Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0759342Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0759361Z 
2023-01-11T21:53:37.0759466Z OK (skipped=1)
2023-01-11T21:53:37.0759485Z 
2023-01-11T21:53:37.0759605Z Generating XML reports...
2023-01-11T21:53:37.0760049Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215120.xml
2023-01-11T21:53:37.0760415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0760586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0760948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0761141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0761160Z 
2023-01-11T21:53:37.0761267Z Running tests...
2023-01-11T21:53:37.0761523Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0761831Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0762091Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:37.0762111Z 
2023-01-11T21:53:37.0762362Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0762470Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0762489Z 
2023-01-11T21:53:37.0762591Z OK (skipped=1)
2023-01-11T21:53:37.0762610Z 
2023-01-11T21:53:37.0762713Z Generating XML reports...
2023-01-11T21:53:37.0763207Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215122.xml
2023-01-11T21:53:37.0763619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0763794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0764169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0764359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0764378Z 
2023-01-11T21:53:37.0764482Z Running tests...
2023-01-11T21:53:37.0764739Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0765051Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0765297Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:37.0765318Z 
2023-01-11T21:53:37.0765578Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0765690Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0765709Z 
2023-01-11T21:53:37.0765812Z OK (skipped=1)
2023-01-11T21:53:37.0765831Z 
2023-01-11T21:53:37.0765951Z Generating XML reports...
2023-01-11T21:53:37.0766397Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215125.xml
2023-01-11T21:53:37.0766765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0766936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0767312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0767484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0767503Z 
2023-01-11T21:53:37.0767613Z Running tests...
2023-01-11T21:53:37.0767877Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0768188Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0768443Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0768659Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29343
2023-01-11T21:53:37.0768876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29344
2023-01-11T21:53:37.0769247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0769401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0769778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0769966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0770336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0770508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0770878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0771064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0771311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0771555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0771937Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0772383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0772659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0772890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0773226Z STAGE:2023-01-11 21:51:32 29343:29343 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0773549Z STAGE:2023-01-11 21:51:32 29344:29344 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0773880Z STAGE:2023-01-11 21:51:33 29343:29343 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0774229Z STAGE:2023-01-11 21:51:33 29343:29343 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0774560Z STAGE:2023-01-11 21:51:33 29344:29344 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0774894Z STAGE:2023-01-11 21:51:33 29344:29344 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0775225Z STAGE:2023-01-11 21:51:33 29343:29343 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0775548Z STAGE:2023-01-11 21:51:33 29344:29344 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0775882Z STAGE:2023-01-11 21:51:33 29344:29344 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0776223Z STAGE:2023-01-11 21:51:33 29344:29344 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0776551Z STAGE:2023-01-11 21:51:33 29343:29343 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0776893Z STAGE:2023-01-11 21:51:33 29343:29343 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0776993Z ok (6.574s)
2023-01-11T21:53:37.0777013Z 
2023-01-11T21:53:37.0777256Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0777374Z Ran 1 test in 6.575s
2023-01-11T21:53:37.0777393Z 
2023-01-11T21:53:37.0777482Z OK
2023-01-11T21:53:37.0777501Z 
2023-01-11T21:53:37.0777631Z Generating XML reports...
2023-01-11T21:53:37.0778083Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215127.xml
2023-01-11T21:53:37.0778451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0778628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0779001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0779189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0779209Z 
2023-01-11T21:53:37.0779298Z Running tests...
2023-01-11T21:53:37.0779557Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0779876Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0780145Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0780366Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29466
2023-01-11T21:53:37.0780583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29467
2023-01-11T21:53:37.0780946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0781118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0781474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0781665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0782108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0782324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0782706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0782893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0783140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0783387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0783785Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0784163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0784400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0784624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0784961Z STAGE:2023-01-11 21:51:41 29467:29467 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0785287Z STAGE:2023-01-11 21:51:41 29466:29466 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0785780Z STAGE:2023-01-11 21:51:42 29466:29466 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0786142Z STAGE:2023-01-11 21:51:42 29466:29466 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0786475Z STAGE:2023-01-11 21:51:42 29467:29467 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0786821Z STAGE:2023-01-11 21:51:42 29467:29467 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0787137Z STAGE:2023-01-11 21:51:42 29467:29467 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0787460Z STAGE:2023-01-11 21:51:42 29466:29466 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0787901Z STAGE:2023-01-11 21:51:42 29467:29467 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0788253Z STAGE:2023-01-11 21:51:42 29467:29467 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0788589Z STAGE:2023-01-11 21:51:42 29466:29466 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0788932Z STAGE:2023-01-11 21:51:42 29466:29466 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0789031Z ok (6.500s)
2023-01-11T21:53:37.0789052Z 
2023-01-11T21:53:37.0789313Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0789422Z Ran 1 test in 6.500s
2023-01-11T21:53:37.0789445Z 
2023-01-11T21:53:37.0789518Z OK
2023-01-11T21:53:37.0789536Z 
2023-01-11T21:53:37.0789660Z Generating XML reports...
2023-01-11T21:53:37.0790114Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215136.xml
2023-01-11T21:53:37.0790482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0790655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0791033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0791223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0791241Z 
2023-01-11T21:53:37.0791346Z Running tests...
2023-01-11T21:53:37.0791589Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0792089Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0792512Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:37.0792534Z 
2023-01-11T21:53:37.0792809Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0792921Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0792939Z 
2023-01-11T21:53:37.0793047Z OK (skipped=1)
2023-01-11T21:53:37.0793066Z 
2023-01-11T21:53:37.0793189Z Generating XML reports...
2023-01-11T21:53:37.0793635Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215145.xml
2023-01-11T21:53:37.0794002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0794159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0794534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0794728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0794747Z 
2023-01-11T21:53:37.0794852Z Running tests...
2023-01-11T21:53:37.0795111Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0795423Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0795681Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T21:53:37.0795700Z 
2023-01-11T21:53:37.0795957Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0796066Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0796085Z 
2023-01-11T21:53:37.0796173Z OK (skipped=1)
2023-01-11T21:53:37.0796192Z 
2023-01-11T21:53:37.0796313Z Generating XML reports...
2023-01-11T21:53:37.0796756Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215148.xml
2023-01-11T21:53:37.0797132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0797306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0797683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0797873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0797893Z 
2023-01-11T21:53:37.0798000Z Running tests...
2023-01-11T21:53:37.0798243Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0798557Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0798939Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T21:53:37.0798961Z 
2023-01-11T21:53:37.0799222Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0799330Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0799349Z 
2023-01-11T21:53:37.0799453Z OK (skipped=1)
2023-01-11T21:53:37.0799472Z 
2023-01-11T21:53:37.0799591Z Generating XML reports...
2023-01-11T21:53:37.0800033Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215150.xml
2023-01-11T21:53:37.0800398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0800555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0800929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0801116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0801135Z 
2023-01-11T21:53:37.0801240Z Running tests...
2023-01-11T21:53:37.0801611Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0801932Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0802196Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... skip: Nccl send/recv tested by test_send_recv_nccl (0.002s)
2023-01-11T21:53:37.0802215Z 
2023-01-11T21:53:37.0802472Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0802581Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0802601Z 
2023-01-11T21:53:37.0802688Z OK (skipped=1)
2023-01-11T21:53:37.0802706Z 
2023-01-11T21:53:37.0802829Z Generating XML reports...
2023-01-11T21:53:37.0803269Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215153.xml
2023-01-11T21:53:37.0803631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0803811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0804184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0804372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0804390Z 
2023-01-11T21:53:37.0804495Z Running tests...
2023-01-11T21:53:37.0804755Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0805048Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0805334Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2023-01-11T21:53:37.0805354Z 
2023-01-11T21:53:37.0805606Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0805713Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0805732Z 
2023-01-11T21:53:37.0805840Z OK (skipped=1)
2023-01-11T21:53:37.0805858Z 
2023-01-11T21:53:37.0805983Z Generating XML reports...
2023-01-11T21:53:37.0806424Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215155.xml
2023-01-11T21:53:37.0806791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0806947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0807325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0807513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0807532Z 
2023-01-11T21:53:37.0807635Z Running tests...
2023-01-11T21:53:37.0807893Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0808204Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0808521Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2023-01-11T21:53:37.0808542Z 
2023-01-11T21:53:37.0808802Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0808910Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0808929Z 
2023-01-11T21:53:37.0809017Z OK (skipped=1)
2023-01-11T21:53:37.0809052Z 
2023-01-11T21:53:37.0809157Z Generating XML reports...
2023-01-11T21:53:37.0809601Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215157.xml
2023-01-11T21:53:37.0809967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0810140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0810516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0810795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0810816Z 
2023-01-11T21:53:37.0810927Z Running tests...
2023-01-11T21:53:37.0811184Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0811479Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0811784Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2023-01-11T21:53:37.0811804Z 
2023-01-11T21:53:37.0812060Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0812169Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0812188Z 
2023-01-11T21:53:37.0812290Z OK (skipped=1)
2023-01-11T21:53:37.0812309Z 
2023-01-11T21:53:37.0812432Z Generating XML reports...
2023-01-11T21:53:37.0812879Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215200.xml
2023-01-11T21:53:37.0813247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0813421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0813779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0813966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0813985Z 
2023-01-11T21:53:37.0814090Z Running tests...
2023-01-11T21:53:37.0814349Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0814658Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0814948Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2023-01-11T21:53:37.0814971Z 
2023-01-11T21:53:37.0815234Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0815342Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0815361Z 
2023-01-11T21:53:37.0815466Z OK (skipped=1)
2023-01-11T21:53:37.0815486Z 
2023-01-11T21:53:37.0815589Z Generating XML reports...
2023-01-11T21:53:37.0816032Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215202.xml
2023-01-11T21:53:37.0816397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0816569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0816945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0817131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0817154Z 
2023-01-11T21:53:37.0817261Z Running tests...
2023-01-11T21:53:37.0817521Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0817814Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0818070Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0818293Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29853
2023-01-11T21:53:37.0818515Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29854
2023-01-11T21:53:37.0818881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0819054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0819426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0819712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0820082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0820235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0820609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0820800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0821046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0821292Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0821692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0822090Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0822320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0822546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0822675Z ok (5.198s)
2023-01-11T21:53:37.0822695Z 
2023-01-11T21:53:37.0822962Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0823073Z Ran 1 test in 5.199s
2023-01-11T21:53:37.0823092Z 
2023-01-11T21:53:37.0823185Z OK
2023-01-11T21:53:37.0823205Z 
2023-01-11T21:53:37.0823326Z Generating XML reports...
2023-01-11T21:53:37.0823774Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215205.xml
2023-01-11T21:53:37.0824138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0824317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0824675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0824864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0824884Z 
2023-01-11T21:53:37.0824989Z Running tests...
2023-01-11T21:53:37.0825246Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0825556Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0825841Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0826060Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29971
2023-01-11T21:53:37.0826275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29972
2023-01-11T21:53:37.0826634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0826809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0827187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0827375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0827732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0827904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0828273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0828457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0828701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0829030Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0829440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0829835Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0830067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0830294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0830629Z STAGE:2023-01-11 21:52:17 29972:29972 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0830955Z STAGE:2023-01-11 21:52:17 29971:29971 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0831297Z STAGE:2023-01-11 21:52:17 29972:29972 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0831626Z STAGE:2023-01-11 21:52:17 29971:29971 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0832134Z STAGE:2023-01-11 21:52:17 29972:29972 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0832500Z STAGE:2023-01-11 21:52:17 29971:29971 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0832604Z ok (5.758s)
2023-01-11T21:53:37.0832625Z 
2023-01-11T21:53:37.0832891Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0832999Z Ran 1 test in 5.758s
2023-01-11T21:53:37.0833018Z 
2023-01-11T21:53:37.0833106Z OK
2023-01-11T21:53:37.0833125Z 
2023-01-11T21:53:37.0833247Z Generating XML reports...
2023-01-11T21:53:37.0833698Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215212.xml
2023-01-11T21:53:37.0834075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0834235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0834615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0834803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0834822Z 
2023-01-11T21:53:37.0834927Z Running tests...
2023-01-11T21:53:37.0835188Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0835502Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0835780Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0835999Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30093
2023-01-11T21:53:37.0836205Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30094
2023-01-11T21:53:37.0836572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0836742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0837118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0837305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0837665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0837838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0838206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0838530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0838763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0839010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0839413Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0839808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0840040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0840271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0840606Z STAGE:2023-01-11 21:52:26 30094:30094 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0840936Z STAGE:2023-01-11 21:52:26 30093:30093 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:53:37.0841256Z STAGE:2023-01-11 21:52:26 30094:30094 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0841592Z STAGE:2023-01-11 21:52:26 30093:30093 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:53:37.0841934Z STAGE:2023-01-11 21:52:26 30094:30094 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0842278Z STAGE:2023-01-11 21:52:26 30093:30093 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:53:37.0842376Z ok (5.773s)
2023-01-11T21:53:37.0842397Z 
2023-01-11T21:53:37.0842657Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0842767Z Ran 1 test in 5.773s
2023-01-11T21:53:37.0842786Z 
2023-01-11T21:53:37.0842875Z OK
2023-01-11T21:53:37.0842894Z 
2023-01-11T21:53:37.0843014Z Generating XML reports...
2023-01-11T21:53:37.0843451Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215221.xml
2023-01-11T21:53:37.0843819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0843992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0844367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0844556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0844575Z 
2023-01-11T21:53:37.0844681Z Running tests...
2023-01-11T21:53:37.0844937Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0845247Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0845530Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2023-01-11T21:53:37.0845556Z 
2023-01-11T21:53:37.0845798Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0845908Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0845927Z 
2023-01-11T21:53:37.0846033Z OK (skipped=1)
2023-01-11T21:53:37.0846053Z 
2023-01-11T21:53:37.0846172Z Generating XML reports...
2023-01-11T21:53:37.0846615Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215229.xml
2023-01-11T21:53:37.0846980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0847153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0847523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0847709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0847778Z 
2023-01-11T21:53:37.0847909Z Running tests...
2023-01-11T21:53:37.0848178Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0848488Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0848762Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2023-01-11T21:53:37.0848781Z 
2023-01-11T21:53:37.0849040Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0849151Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0849170Z 
2023-01-11T21:53:37.0849275Z OK (skipped=1)
2023-01-11T21:53:37.0849295Z 
2023-01-11T21:53:37.0849414Z Generating XML reports...
2023-01-11T21:53:37.0849844Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215231.xml
2023-01-11T21:53:37.0850217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0850396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0850774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0850963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0850982Z 
2023-01-11T21:53:37.0851087Z Running tests...
2023-01-11T21:53:37.0851348Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0851660Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0851959Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2023-01-11T21:53:37.0851978Z 
2023-01-11T21:53:37.0852218Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0852332Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0852354Z 
2023-01-11T21:53:37.0852460Z OK (skipped=1)
2023-01-11T21:53:37.0852478Z 
2023-01-11T21:53:37.0852600Z Generating XML reports...
2023-01-11T21:53:37.0853043Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215234.xml
2023-01-11T21:53:37.0853412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0853586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0853957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0854144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0854163Z 
2023-01-11T21:53:37.0854251Z Running tests...
2023-01-11T21:53:37.0854511Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0854827Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0855120Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2023-01-11T21:53:37.0855140Z 
2023-01-11T21:53:37.0855396Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0855504Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0855523Z 
2023-01-11T21:53:37.0855627Z OK (skipped=1)
2023-01-11T21:53:37.0855645Z 
2023-01-11T21:53:37.0855766Z Generating XML reports...
2023-01-11T21:53:37.0856211Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215236.xml
2023-01-11T21:53:37.0856563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0856735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0857208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0857405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0857424Z 
2023-01-11T21:53:37.0857531Z Running tests...
2023-01-11T21:53:37.0857792Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0858102Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0858386Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2023-01-11T21:53:37.0858405Z 
2023-01-11T21:53:37.0858662Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0858754Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0858772Z 
2023-01-11T21:53:37.0858878Z OK (skipped=1)
2023-01-11T21:53:37.0858897Z 
2023-01-11T21:53:37.0859020Z Generating XML reports...
2023-01-11T21:53:37.0859469Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215238.xml
2023-01-11T21:53:37.0859836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0860009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0860383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0860569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0860588Z 
2023-01-11T21:53:37.0860676Z Running tests...
2023-01-11T21:53:37.0860935Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0861244Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0861535Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2023-01-11T21:53:37.0861558Z 
2023-01-11T21:53:37.0861815Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0861923Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0861942Z 
2023-01-11T21:53:37.0862046Z OK (skipped=1)
2023-01-11T21:53:37.0862064Z 
2023-01-11T21:53:37.0862184Z Generating XML reports...
2023-01-11T21:53:37.0862623Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215241.xml
2023-01-11T21:53:37.0862971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0863146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0863519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0863714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0863733Z 
2023-01-11T21:53:37.0863838Z Running tests...
2023-01-11T21:53:37.0864095Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0864402Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0864669Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0864891Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30413
2023-01-11T21:53:37.0865090Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30414
2023-01-11T21:53:37.0865454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0865626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0866049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0866306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0866669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0866843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0867217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0867386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0867634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0867876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0868275Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0868680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0868911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0869137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0869236Z ok (6.107s)
2023-01-11T21:53:37.0869255Z 
2023-01-11T21:53:37.0869516Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0869608Z Ran 1 test in 6.107s
2023-01-11T21:53:37.0869627Z 
2023-01-11T21:53:37.0869721Z OK
2023-01-11T21:53:37.0869740Z 
2023-01-11T21:53:37.0869857Z Generating XML reports...
2023-01-11T21:53:37.0870301Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215243.xml
2023-01-11T21:53:37.0870667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0870845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0871220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0871407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0871426Z 
2023-01-11T21:53:37.0871535Z Running tests...
2023-01-11T21:53:37.0871775Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0872284Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0872565Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2023-01-11T21:53:37.0872585Z 
2023-01-11T21:53:37.0872848Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0872963Z Ran 1 test in 0.002s
2023-01-11T21:53:37.0872986Z 
2023-01-11T21:53:37.0873093Z OK (skipped=1)
2023-01-11T21:53:37.0873112Z 
2023-01-11T21:53:37.0873233Z Generating XML reports...
2023-01-11T21:53:37.0873680Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215252.xml
2023-01-11T21:53:37.0874031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0874205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0874579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0874767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0874786Z 
2023-01-11T21:53:37.0874892Z Running tests...
2023-01-11T21:53:37.0875148Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0875602Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0875867Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0876090Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30565
2023-01-11T21:53:37.0876290Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30566
2023-01-11T21:53:37.0876663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0876835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0877208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0877399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0877765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0877938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0878307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0878477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0878721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0878963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0879358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0879748Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0879983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0880210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0880310Z ok (5.684s)
2023-01-11T21:53:37.0880329Z 
2023-01-11T21:53:37.0880590Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0880682Z Ran 1 test in 5.684s
2023-01-11T21:53:37.0880701Z 
2023-01-11T21:53:37.0880790Z OK
2023-01-11T21:53:37.0880809Z 
2023-01-11T21:53:37.0880930Z Generating XML reports...
2023-01-11T21:53:37.0881376Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215254.xml
2023-01-11T21:53:37.0881741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0881918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0882296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0882488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0882508Z 
2023-01-11T21:53:37.0882614Z Running tests...
2023-01-11T21:53:37.0882861Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0883174Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0883470Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0883689Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30680
2023-01-11T21:53:37.0883903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30681
2023-01-11T21:53:37.0884271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0884538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0884922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0885093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0885455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0885628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0885997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0886179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0886424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0886668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0887070Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0887465Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0887677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0887905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0888688Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T21:53:37.0889473Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T21:53:37.0889574Z ok (6.194s)
2023-01-11T21:53:37.0889594Z 
2023-01-11T21:53:37.0889857Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0889966Z Ran 1 test in 6.194s
2023-01-11T21:53:37.0889986Z 
2023-01-11T21:53:37.0890074Z OK
2023-01-11T21:53:37.0890093Z 
2023-01-11T21:53:37.0890216Z Generating XML reports...
2023-01-11T21:53:37.0890677Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215302.xml
2023-01-11T21:53:37.0891043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0891219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0891584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0891775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0891794Z 
2023-01-11T21:53:37.0891899Z Running tests...
2023-01-11T21:53:37.0892155Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0892468Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0892802Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0893072Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30799
2023-01-11T21:53:37.0893288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30800
2023-01-11T21:53:37.0893638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0893814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0894189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0894380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0894739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0894909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0895284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0895468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0895713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0895940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0896340Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0896737Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0896965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0897194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0897440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:53:37.0897683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:53:37.0898075Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:37.0898465Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:37.0898687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:53:37.0898922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:53:37.0899308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:53:37.0899709Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:53:37.0899810Z ok (10.987s)
2023-01-11T21:53:37.0899830Z 
2023-01-11T21:53:37.0900092Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0900200Z Ran 1 test in 10.987s
2023-01-11T21:53:37.0900219Z 
2023-01-11T21:53:37.0900308Z OK
2023-01-11T21:53:37.0900327Z 
2023-01-11T21:53:37.0900447Z Generating XML reports...
2023-01-11T21:53:37.0900880Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215311.xml
2023-01-11T21:53:37.0901246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0901414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0901791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0902070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0902091Z 
2023-01-11T21:53:37.0902203Z Running tests...
2023-01-11T21:53:37.0902464Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0902775Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T21:53:37.0903047Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:53:37.0903265Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30921
2023-01-11T21:53:37.0903480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30922
2023-01-11T21:53:37.0903845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0904016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0904398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0904590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0904949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:53:37.0905117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:53:37.0905472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:53:37.0905658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:53:37.0905901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:53:37.0906145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:53:37.0906548Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0906942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:53:37.0907173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:53:37.0907397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:53:37.0907635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:53:37.0907858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:53:37.0908257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:37.0908648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:53:37.0908894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:53:37.0909131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:53:37.0909517Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:53:37.0909909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:53:37.0910008Z ok (10.978s)
2023-01-11T21:53:37.0910027Z 
2023-01-11T21:53:37.0910287Z ----------------------------------------------------------------------
2023-01-11T21:53:37.0910380Z Ran 1 test in 10.978s
2023-01-11T21:53:37.0910398Z 
2023-01-11T21:53:37.0910488Z OK
2023-01-11T21:53:37.0910507Z 
2023-01-11T21:53:37.0910625Z Generating XML reports...
2023-01-11T21:53:37.0911173Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215325.xml
2023-01-11T21:53:37.0911194Z 
2023-01-11T21:53:37.0911624Z ##[endgroup]
2023-01-11T21:53:37.0912280Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_qma1brgt)
2023-01-11T21:53:37.0912301Z 
2023-01-11T21:53:37.0912518Z Running distributed tests for the nccl backend with file init_method in shard 2 of 3
2023-01-11T21:53:37.0913042Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:53:36.887931]
2023-01-11T22:18:07.0267988Z 
2023-01-11T22:18:07.0270724Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2023-01-11T22:18:07.0272591Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_qe58onb1)
2023-01-11T22:18:07.0273154Z <unittest.suite.TestSuite tests=[]>
2023-01-11T22:18:07.0311690Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward_grad_as_bucket_view_false>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward_ignored_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2023-01-11T22:18:07.0350986Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0351673Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0352800Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0353208Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0353671Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0354153Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0354664Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0355147Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0355869Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0356428Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0357003Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0357517Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0358239Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0358773Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0359414Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0359936Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0360406Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0360838Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0361250Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0361698Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0362182Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0362828Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0363248Z test_all_gather (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0363642Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0364277Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0364691Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0365118Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0365542Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0365936Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0366325Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0366717Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0367118Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0367487Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0367894Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0368320Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0368970Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0369543Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0369970Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0370481Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0370955Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0371364Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0371809Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0372232Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0372685Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0373309Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0373741Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0374177Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0374612Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0375031Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0375462Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0375899Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0376310Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0376705Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0377145Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0377760Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0378174Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0378582Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0378995Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0379398Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0379776Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0380176Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0380572Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0380930Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0381305Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0381689Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0382092Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0382476Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0382884Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0383268Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0383631Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0384023Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0384415Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0384793Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0385201Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0385584Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0385959Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0386318Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0386706Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0387429Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0387818Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0388541Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0388949Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0389340Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0389773Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0390399Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0390857Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0391290Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0391745Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0392824Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0393442Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0393878Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0394315Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0394758Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0395191Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0395648Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0396113Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0396564Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0397007Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0397431Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0397829Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0398196Z test_backend_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0398559Z test_barrier (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0399122Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0399483Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0399882Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0400274Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0400659Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0401077Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0401640Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0402053Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0402438Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0402842Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0403455Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0403882Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0404282Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0404697Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0405111Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0405522Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0405957Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0406530Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0406905Z test_broadcast (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0407282Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0407676Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0408070Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0408446Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0408842Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0409300Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0409795Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0410263Z test_ddp_apply_optim_in_backward (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0410728Z test_ddp_apply_optim_in_backward_grad_as_bucket_view_false (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0411212Z test_ddp_apply_optim_in_backward_ignored_params (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0411631Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0412047Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0412466Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0412882Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0413331Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0413802Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0414247Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0414658Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0415108Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0415519Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0415928Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0416478Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0416898Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0417317Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0417735Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0418173Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0418591Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0418991Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0419458Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0420174Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0420744Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0421340Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0421972Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0422578Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0423187Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0424112Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0424722Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0425322Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0425872Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0426356Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0426800Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0427191Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0427592Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0427994Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0428394Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0428814Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0429242Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0429703Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0430176Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0430762Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0431216Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0431633Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0432636Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0433104Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0433523Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0433937Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0434351Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0434781Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0435202Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0435609Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0435999Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0436415Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0436816Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0437226Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0437690Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0438126Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0438504Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0438888Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0439288Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0439711Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0440109Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0440489Z test_gather (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0440854Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0441203Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0441732Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0442120Z test_gather_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0442609Z test_gather_object (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0443052Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0443437Z test_get_backend (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0443801Z test_get_future (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0444139Z test_get_rank (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0444519Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0444912Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0445281Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0445646Z test_irecv (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0445991Z test_isend (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0446363Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0446764Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0447178Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0447633Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0448067Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0448482Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0448910Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0449323Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0449750Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0450171Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0450600Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0450997Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0451616Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0452370Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0453095Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0453850Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0454770Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0455765Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0456724Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0457642Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0458574Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0459414Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0460245Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0461047Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0461836Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0462727Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0463627Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0464568Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0465543Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0466754Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0467603Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0468365Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0469169Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0469969Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0470765Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0471493Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0472853Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0473640Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0474362Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0475070Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0475809Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0476569Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0477385Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0478212Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0478965Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0479691Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0480477Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0481267Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0481977Z test_scatter (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0482695Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0483445Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0484168Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0484930Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0485723Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0486461Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0487200Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0487935Z test_send_recv (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0488721Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0489552Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0490453Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0491321Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0492092Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0492906Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0493785Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0494645Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0495416Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0496243Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0497098Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0497907Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0498716Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0499523Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0500285Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0501069Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0501945Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0503127Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0504008Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.0505371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0506289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0507450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0508387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0508815Z 
2023-01-11T22:18:07.0509034Z Running tests...
2023-01-11T22:18:07.0509830Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0510886Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.0512649Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.0513791Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31076
2023-01-11T22:18:07.0514674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31077
2023-01-11T22:18:07.0515969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0516868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0518050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0518966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0520138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0521069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0522235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0523171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0524047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.0524974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.0526281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.0527669Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.0528734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.0529700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.0530775Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T22:18:07.0532514Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T22:18:07.0533874Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T22:18:07.0535601Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T22:18:07.0536934Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T22:18:07.0538861Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T22:18:07.0540341Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T22:18:07.0542054Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T22:18:07.0543373Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T22:18:07.0545133Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T22:18:07.0546484Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T22:18:07.0548206Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T22:18:07.0549559Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T22:18:07.0551283Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T22:18:07.0553268Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T22:18:07.0554993Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T22:18:07.0555959Z ok (5.745s)
2023-01-11T22:18:07.0556256Z 
2023-01-11T22:18:07.0556796Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0557441Z Ran 1 test in 5.745s
2023-01-11T22:18:07.0557766Z 
2023-01-11T22:18:07.0557938Z OK
2023-01-11T22:18:07.0558169Z 
2023-01-11T22:18:07.0558411Z Generating XML reports...
2023-01-11T22:18:07.0559649Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215341.xml
2023-01-11T22:18:07.0561126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0562028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0563229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0564209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0564662Z 
2023-01-11T22:18:07.0564865Z Running tests...
2023-01-11T22:18:07.0565658Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0566741Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.0567819Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2023-01-11T22:18:07.0568450Z 
2023-01-11T22:18:07.0568996Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0569615Z Ran 1 test in 0.004s
2023-01-11T22:18:07.0569919Z 
2023-01-11T22:18:07.0570122Z OK (skipped=1)
2023-01-11T22:18:07.0570439Z 
2023-01-11T22:18:07.0570673Z Generating XML reports...
2023-01-11T22:18:07.0571889Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215349.xml
2023-01-11T22:18:07.0573465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0574374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0575716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0576766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0577225Z 
2023-01-11T22:18:07.0577430Z Running tests...
2023-01-11T22:18:07.0578243Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0579296Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.0580295Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.0581290Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31224
2023-01-11T22:18:07.0582212Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31225
2023-01-11T22:18:07.0583431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0584336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0585549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0586488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0587706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0588605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0589779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0590716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0591651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.0593100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.0594493Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.0595932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.0597006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.0597993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.0598669Z ok (4.276s)
2023-01-11T22:18:07.0598967Z 
2023-01-11T22:18:07.0599514Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0600179Z Ran 1 test in 4.276s
2023-01-11T22:18:07.0600493Z 
2023-01-11T22:18:07.0600663Z OK
2023-01-11T22:18:07.0600900Z 
2023-01-11T22:18:07.0601135Z Generating XML reports...
2023-01-11T22:18:07.0602384Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215351.xml
2023-01-11T22:18:07.0603850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0604741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0605925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0606884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0607345Z 
2023-01-11T22:18:07.0607561Z Running tests...
2023-01-11T22:18:07.0608345Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0609410Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.0610487Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.0612794Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.668s)
2023-01-11T22:18:07.0613960Z 
2023-01-11T22:18:07.0614521Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0615134Z Ran 1 test in 1.668s
2023-01-11T22:18:07.0615448Z 
2023-01-11T22:18:07.0615655Z OK (skipped=1)
2023-01-11T22:18:07.0615949Z 
2023-01-11T22:18:07.0616188Z Generating XML reports...
2023-01-11T22:18:07.0617412Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215358.xml
2023-01-11T22:18:07.0618867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0619758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0620932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0621848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0622304Z 
2023-01-11T22:18:07.0622506Z Running tests...
2023-01-11T22:18:07.0623284Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0624278Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.0625332Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2023-01-11T22:18:07.0625997Z 
2023-01-11T22:18:07.0626541Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0627169Z Ran 1 test in 0.002s
2023-01-11T22:18:07.0627451Z 
2023-01-11T22:18:07.0627665Z OK (skipped=1)
2023-01-11T22:18:07.0627958Z 
2023-01-11T22:18:07.0628208Z Generating XML reports...
2023-01-11T22:18:07.0629454Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215402.xml
2023-01-11T22:18:07.0630938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0632326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0633535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0634483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0634954Z 
2023-01-11T22:18:07.0635147Z Running tests...
2023-01-11T22:18:07.0635951Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0637028Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.0638174Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2023-01-11T22:18:07.0638868Z 
2023-01-11T22:18:07.0639397Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0640044Z Ran 1 test in 0.002s
2023-01-11T22:18:07.0640354Z 
2023-01-11T22:18:07.0640555Z OK (skipped=1)
2023-01-11T22:18:07.0640846Z 
2023-01-11T22:18:07.0641073Z Generating XML reports...
2023-01-11T22:18:07.0642294Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215405.xml
2023-01-11T22:18:07.0643756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0644651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0645814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0647087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0647563Z 
2023-01-11T22:18:07.0647779Z Running tests...
2023-01-11T22:18:07.0648552Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0649627Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.0650754Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.0651824Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31429
2023-01-11T22:18:07.0652705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31430
2023-01-11T22:18:07.0653967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0654864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0656026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0657008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0658166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0659065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0660189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0661142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0662060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.0663102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.0664436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.0665848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.0666907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.0667831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.0668816Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0669810Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0670792Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0671746Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0673356Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0674352Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0675298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0676307Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0677329Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0678327Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0679333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0680336Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0681046Z ok (6.700s)
2023-01-11T22:18:07.0681357Z 
2023-01-11T22:18:07.0681923Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0682797Z Ran 1 test in 6.700s
2023-01-11T22:18:07.0683233Z 
2023-01-11T22:18:07.0683415Z OK
2023-01-11T22:18:07.0683673Z 
2023-01-11T22:18:07.0683909Z Generating XML reports...
2023-01-11T22:18:07.0685115Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215407.xml
2023-01-11T22:18:07.0686592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0687496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0688650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0689600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0690075Z 
2023-01-11T22:18:07.0690284Z Running tests...
2023-01-11T22:18:07.0691102Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0692192Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.0693345Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.0694442Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31548
2023-01-11T22:18:07.0695351Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31549
2023-01-11T22:18:07.0696581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0697485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0698668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0699601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0700809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0701726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0702916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0703828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0704744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.0705744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.0707108Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.0708567Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.0709661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.0710625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.0711567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0712963Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0713972Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0714949Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0715626Z ok (5.784s)
2023-01-11T22:18:07.0715906Z 
2023-01-11T22:18:07.0716453Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0717107Z Ran 1 test in 5.784s
2023-01-11T22:18:07.0717424Z 
2023-01-11T22:18:07.0717572Z OK
2023-01-11T22:18:07.0717836Z 
2023-01-11T22:18:07.0718281Z Generating XML reports...
2023-01-11T22:18:07.0719665Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215416.xml
2023-01-11T22:18:07.0721180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0722059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0723278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0724237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0724691Z 
2023-01-11T22:18:07.0724874Z Running tests...
2023-01-11T22:18:07.0725710Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0726779Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.0727974Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.0729091Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31667
2023-01-11T22:18:07.0730005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31668
2023-01-11T22:18:07.0731336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0732225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0733401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0734354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0735554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0736427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0737609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0738542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0739480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.0740460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.0741812Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.0743261Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.0744278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.0745245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.0746233Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0747210Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0748161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0749144Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0750102Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0751046Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0752435Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0753412Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0754311Z ok (5.794s)
2023-01-11T22:18:07.0754597Z 
2023-01-11T22:18:07.0755277Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0755931Z Ran 1 test in 5.794s
2023-01-11T22:18:07.0756249Z 
2023-01-11T22:18:07.0756427Z OK
2023-01-11T22:18:07.0756688Z 
2023-01-11T22:18:07.0756903Z Generating XML reports...
2023-01-11T22:18:07.0758162Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215425.xml
2023-01-11T22:18:07.0759623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0760500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0761649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0762573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0763054Z 
2023-01-11T22:18:07.0763260Z Running tests...
2023-01-11T22:18:07.0764046Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0765114Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.0766292Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.0767426Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31786
2023-01-11T22:18:07.0768316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31787
2023-01-11T22:18:07.0769573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0770487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0771665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0772647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0773844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0774763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0775909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0776870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0777803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.0778822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.0780150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.0781603Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.0782667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.0783637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.0784600Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0785584Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0786297Z ok (5.794s)
2023-01-11T22:18:07.0786572Z 
2023-01-11T22:18:07.0787085Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0787711Z Ran 1 test in 5.794s
2023-01-11T22:18:07.0788024Z 
2023-01-11T22:18:07.0788193Z OK
2023-01-11T22:18:07.0788447Z 
2023-01-11T22:18:07.0788688Z Generating XML reports...
2023-01-11T22:18:07.0790173Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215433.xml
2023-01-11T22:18:07.0791687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0792958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0794109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0795080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0795536Z 
2023-01-11T22:18:07.0795743Z Running tests...
2023-01-11T22:18:07.0796567Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0797615Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.0798819Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.0799995Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31905
2023-01-11T22:18:07.0800894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31906
2023-01-11T22:18:07.0802158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0803081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0804279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0805209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0806437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0807339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0808581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0809505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0810431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.0811436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.0812762Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.0814158Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.0815206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.0816163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.0817165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0818166Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0819139Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0820094Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0820768Z ok (6.487s)
2023-01-11T22:18:07.0821055Z 
2023-01-11T22:18:07.0821587Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0822227Z Ran 1 test in 6.487s
2023-01-11T22:18:07.0822509Z 
2023-01-11T22:18:07.0822677Z OK
2023-01-11T22:18:07.0822935Z 
2023-01-11T22:18:07.0823175Z Generating XML reports...
2023-01-11T22:18:07.0824415Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215441.xml
2023-01-11T22:18:07.0826139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0827070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0828267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0829211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0829683Z 
2023-01-11T22:18:07.0829853Z Running tests...
2023-01-11T22:18:07.0830704Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0831789Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.0833299Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.0834350Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32024
2023-01-11T22:18:07.0835278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32025
2023-01-11T22:18:07.0836551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0837420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0838604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0839560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0840768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0841650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0842817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0843801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0844681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.0845675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.0847052Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.0848464Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.0849503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.0850464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.0851411Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0852433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0853390Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0854376Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0855069Z ok (6.191s)
2023-01-11T22:18:07.0855356Z 
2023-01-11T22:18:07.0855882Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0856515Z Ran 1 test in 6.191s
2023-01-11T22:18:07.0856837Z 
2023-01-11T22:18:07.0857005Z OK
2023-01-11T22:18:07.0857259Z 
2023-01-11T22:18:07.0857494Z Generating XML reports...
2023-01-11T22:18:07.0858728Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215450.xml
2023-01-11T22:18:07.0860184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0861272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0862567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0863546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0864009Z 
2023-01-11T22:18:07.0864229Z Running tests...
2023-01-11T22:18:07.0865031Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0866063Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.0867242Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.0868402Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32143
2023-01-11T22:18:07.0869313Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32144
2023-01-11T22:18:07.0870594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0871490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0872963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0873932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0875113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0876049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0877244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0878182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0879098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.0880117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.0881448Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.0882878Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.0883930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.0884882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.0885839Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0886840Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0887809Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0888824Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.0889508Z ok (5.645s)
2023-01-11T22:18:07.0889796Z 
2023-01-11T22:18:07.0890347Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0890986Z Ran 1 test in 5.645s
2023-01-11T22:18:07.0891272Z 
2023-01-11T22:18:07.0891447Z OK
2023-01-11T22:18:07.0891705Z 
2023-01-11T22:18:07.0891940Z Generating XML reports...
2023-01-11T22:18:07.0893194Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215459.xml
2023-01-11T22:18:07.0894638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0895550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0896911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0897971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0898433Z 
2023-01-11T22:18:07.0898608Z Running tests...
2023-01-11T22:18:07.0899424Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0900490Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.0901610Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.0903781Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.681s)
2023-01-11T22:18:07.0904844Z 
2023-01-11T22:18:07.0905408Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0906057Z Ran 1 test in 1.681s
2023-01-11T22:18:07.0906375Z 
2023-01-11T22:18:07.0906571Z OK (skipped=1)
2023-01-11T22:18:07.0906836Z 
2023-01-11T22:18:07.0907080Z Generating XML reports...
2023-01-11T22:18:07.0908315Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215507.xml
2023-01-11T22:18:07.0909804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0910673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0912199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0913172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0913633Z 
2023-01-11T22:18:07.0913846Z Running tests...
2023-01-11T22:18:07.0914657Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0915736Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.0916867Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.0917934Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32296
2023-01-11T22:18:07.0918851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32297
2023-01-11T22:18:07.0920094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0921009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0922159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0923117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0924307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0925219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0926372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0927314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0928227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.0929207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.0930553Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.0932058Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.0933413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.0934380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.0935068Z ok (4.384s)
2023-01-11T22:18:07.0935355Z 
2023-01-11T22:18:07.0935893Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0936507Z Ran 1 test in 4.384s
2023-01-11T22:18:07.0936825Z 
2023-01-11T22:18:07.0936997Z OK
2023-01-11T22:18:07.0937252Z 
2023-01-11T22:18:07.0937487Z Generating XML reports...
2023-01-11T22:18:07.0938744Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215511.xml
2023-01-11T22:18:07.0940192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0941121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0942321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0943292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0943734Z 
2023-01-11T22:18:07.0943945Z Running tests...
2023-01-11T22:18:07.0944761Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0945859Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.0946980Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.0949192Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.662s)
2023-01-11T22:18:07.0950290Z 
2023-01-11T22:18:07.0950834Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0951467Z Ran 1 test in 1.662s
2023-01-11T22:18:07.0951779Z 
2023-01-11T22:18:07.0952316Z OK (skipped=1)
2023-01-11T22:18:07.0952633Z 
2023-01-11T22:18:07.0952871Z Generating XML reports...
2023-01-11T22:18:07.0954106Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215518.xml
2023-01-11T22:18:07.0955608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0956490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0957664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0958637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0959116Z 
2023-01-11T22:18:07.0959305Z Running tests...
2023-01-11T22:18:07.0960119Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0961211Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.0962284Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.0963303Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32435
2023-01-11T22:18:07.0964232Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32436
2023-01-11T22:18:07.0965500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0966398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0967599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0968857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0970097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0970997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0972196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0973147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0974075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.0975037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.0976410Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.0977801Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.0978822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.0979785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.0980473Z ok (5.557s)
2023-01-11T22:18:07.0980749Z 
2023-01-11T22:18:07.0981285Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0981883Z Ran 1 test in 5.557s
2023-01-11T22:18:07.0982189Z 
2023-01-11T22:18:07.0982358Z OK
2023-01-11T22:18:07.0982606Z 
2023-01-11T22:18:07.0982845Z Generating XML reports...
2023-01-11T22:18:07.0984043Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215523.xml
2023-01-11T22:18:07.0985521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0986439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.0987645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.0988579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.0989041Z 
2023-01-11T22:18:07.0989247Z Running tests...
2023-01-11T22:18:07.0990071Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0991132Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.0992454Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2023-01-11T22:18:07.0993058Z 
2023-01-11T22:18:07.0993614Z ----------------------------------------------------------------------
2023-01-11T22:18:07.0994257Z Ran 1 test in 0.002s
2023-01-11T22:18:07.0994587Z 
2023-01-11T22:18:07.0994771Z OK (skipped=1)
2023-01-11T22:18:07.0995073Z 
2023-01-11T22:18:07.0995316Z Generating XML reports...
2023-01-11T22:18:07.0996580Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215531.xml
2023-01-11T22:18:07.0998054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.0998956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1000124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1001073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1001560Z 
2023-01-11T22:18:07.1001756Z Running tests...
2023-01-11T22:18:07.1002591Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1003691Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1004939Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.1005906Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1006888Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32583
2023-01-11T22:18:07.1007828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32584
2023-01-11T22:18:07.1009191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1010113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1011319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1012269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1013450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1014350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1015534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1016454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1017358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1018379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1019719Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1021105Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1022202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1023145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1023846Z ok (6.051s)
2023-01-11T22:18:07.1024103Z 
2023-01-11T22:18:07.1024648Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1025298Z Ran 1 test in 6.052s
2023-01-11T22:18:07.1025605Z 
2023-01-11T22:18:07.1025770Z OK
2023-01-11T22:18:07.1026018Z 
2023-01-11T22:18:07.1026236Z Generating XML reports...
2023-01-11T22:18:07.1027470Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215533.xml
2023-01-11T22:18:07.1028969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1029882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1031151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1032362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1032824Z 
2023-01-11T22:18:07.1033046Z Running tests...
2023-01-11T22:18:07.1033838Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1034906Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1035862Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.1036870Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1037856Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32702
2023-01-11T22:18:07.1038771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32703
2023-01-11T22:18:07.1040372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1041273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1042460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1043390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1044583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1045447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1046601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1047562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1048451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1049455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1050805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1052213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1053238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1054195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1055171Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.1056158Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.1056811Z ok (6.053s)
2023-01-11T22:18:07.1057120Z 
2023-01-11T22:18:07.1057688Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1058327Z Ran 1 test in 6.054s
2023-01-11T22:18:07.1058642Z 
2023-01-11T22:18:07.1058779Z OK
2023-01-11T22:18:07.1059033Z 
2023-01-11T22:18:07.1059265Z Generating XML reports...
2023-01-11T22:18:07.1060513Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215542.xml
2023-01-11T22:18:07.1062029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1062933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1064116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1065075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1065543Z 
2023-01-11T22:18:07.1065728Z Running tests...
2023-01-11T22:18:07.1066558Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1067646Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1068645Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.1069715Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1070738Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32821
2023-01-11T22:18:07.1071663Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32822
2023-01-11T22:18:07.1073278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1074191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1075522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1076588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1077769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1078695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1079865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1080837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1081707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1082693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1084072Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1085485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1086552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1087513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1088466Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.1089404Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.1090101Z ok (6.049s)
2023-01-11T22:18:07.1090392Z 
2023-01-11T22:18:07.1090937Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1091554Z Ran 1 test in 6.050s
2023-01-11T22:18:07.1091866Z 
2023-01-11T22:18:07.1092038Z OK
2023-01-11T22:18:07.1092289Z 
2023-01-11T22:18:07.1092545Z Generating XML reports...
2023-01-11T22:18:07.1093769Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215550.xml
2023-01-11T22:18:07.1095225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1096134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1097304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1098258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1098698Z 
2023-01-11T22:18:07.1098912Z Running tests...
2023-01-11T22:18:07.1099734Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1100828Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1101791Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T22:18:07.1102807Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1103799Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32940
2023-01-11T22:18:07.1104708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32941
2023-01-11T22:18:07.1105962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1106874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1108070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1109006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1110206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1111353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1113130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1114084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1115007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1116012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1117345Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1118789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1119869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1120865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1121552Z ok (6.191s)
2023-01-11T22:18:07.1121842Z 
2023-01-11T22:18:07.1122384Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1123031Z Ran 1 test in 6.191s
2023-01-11T22:18:07.1123345Z 
2023-01-11T22:18:07.1123516Z OK
2023-01-11T22:18:07.1123748Z 
2023-01-11T22:18:07.1123990Z Generating XML reports...
2023-01-11T22:18:07.1125242Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215559.xml
2023-01-11T22:18:07.1126734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1127635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1128811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1129789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1130253Z 
2023-01-11T22:18:07.1130461Z Running tests...
2023-01-11T22:18:07.1131320Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1132397Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1133423Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.1134003Z 
2023-01-11T22:18:07.1134503Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1135147Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1135460Z 
2023-01-11T22:18:07.1135666Z OK (skipped=1)
2023-01-11T22:18:07.1135966Z 
2023-01-11T22:18:07.1136204Z Generating XML reports...
2023-01-11T22:18:07.1137416Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215607.xml
2023-01-11T22:18:07.1138908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1139817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1140992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1141956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1142412Z 
2023-01-11T22:18:07.1142623Z Running tests...
2023-01-11T22:18:07.1143432Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1144468Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1145555Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2023-01-11T22:18:07.1146197Z 
2023-01-11T22:18:07.1147081Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1147744Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1148038Z 
2023-01-11T22:18:07.1148236Z OK (skipped=1)
2023-01-11T22:18:07.1148528Z 
2023-01-11T22:18:07.1148757Z Generating XML reports...
2023-01-11T22:18:07.1149984Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215610.xml
2023-01-11T22:18:07.1151413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1152679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1153882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1154835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1155282Z 
2023-01-11T22:18:07.1155482Z Running tests...
2023-01-11T22:18:07.1156315Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1157410Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1158481Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2023-01-11T22:18:07.1159136Z 
2023-01-11T22:18:07.1159679Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1160324Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1160641Z 
2023-01-11T22:18:07.1160849Z OK (skipped=1)
2023-01-11T22:18:07.1161124Z 
2023-01-11T22:18:07.1161366Z Generating XML reports...
2023-01-11T22:18:07.1162595Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215612.xml
2023-01-11T22:18:07.1164068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1164971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1166161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1167123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1167592Z 
2023-01-11T22:18:07.1167796Z Running tests...
2023-01-11T22:18:07.1168608Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1169689Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1170797Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2023-01-11T22:18:07.1171445Z 
2023-01-11T22:18:07.1172006Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1172618Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1172928Z 
2023-01-11T22:18:07.1173146Z OK (skipped=1)
2023-01-11T22:18:07.1173533Z 
2023-01-11T22:18:07.1173785Z Generating XML reports...
2023-01-11T22:18:07.1175025Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215615.xml
2023-01-11T22:18:07.1176486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1177397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1178593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1179509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1179969Z 
2023-01-11T22:18:07.1180179Z Running tests...
2023-01-11T22:18:07.1180985Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1182042Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1183439Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2023-01-11T22:18:07.1184113Z 
2023-01-11T22:18:07.1184651Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1185294Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1185584Z 
2023-01-11T22:18:07.1185779Z OK (skipped=1)
2023-01-11T22:18:07.1186067Z 
2023-01-11T22:18:07.1186307Z Generating XML reports...
2023-01-11T22:18:07.1187563Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215617.xml
2023-01-11T22:18:07.1189032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1189943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1191153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1192398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1192866Z 
2023-01-11T22:18:07.1193052Z Running tests...
2023-01-11T22:18:07.1193890Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1194967Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1196077Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support all_gather_coalesced (0.002s)
2023-01-11T22:18:07.1196699Z 
2023-01-11T22:18:07.1197245Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1197890Z Ran 1 test in 0.003s
2023-01-11T22:18:07.1198202Z 
2023-01-11T22:18:07.1198402Z OK (skipped=1)
2023-01-11T22:18:07.1198694Z 
2023-01-11T22:18:07.1198903Z Generating XML reports...
2023-01-11T22:18:07.1200126Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215619.xml
2023-01-11T22:18:07.1201630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1202544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1203712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1204660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1205112Z 
2023-01-11T22:18:07.1205324Z Running tests...
2023-01-11T22:18:07.1206108Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1207195Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1208224Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.1208829Z 
2023-01-11T22:18:07.1209384Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1209996Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1210313Z 
2023-01-11T22:18:07.1210521Z OK (skipped=1)
2023-01-11T22:18:07.1210810Z 
2023-01-11T22:18:07.1211048Z Generating XML reports...
2023-01-11T22:18:07.1212253Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215622.xml
2023-01-11T22:18:07.1213705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1214609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1215797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1216728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1217205Z 
2023-01-11T22:18:07.1217617Z Running tests...
2023-01-11T22:18:07.1218549Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1219621Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1220635Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1221618Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33290
2023-01-11T22:18:07.1222544Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33291
2023-01-11T22:18:07.1223794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1224714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1225889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1226846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1228033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1228941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1230131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1231137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1232325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1233329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1234697Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1236102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1237198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1238165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1239359Z STAGE:2023-01-11 21:56:29 33290:33290 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1240517Z STAGE:2023-01-11 21:56:29 33291:33291 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1241693Z STAGE:2023-01-11 21:56:30 33290:33290 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1242880Z STAGE:2023-01-11 21:56:30 33291:33291 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1244065Z STAGE:2023-01-11 21:56:30 33290:33290 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1245316Z STAGE:2023-01-11 21:56:30 33291:33291 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1246524Z STAGE:2023-01-11 21:56:30 33290:33290 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1247705Z STAGE:2023-01-11 21:56:30 33290:33290 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1248891Z STAGE:2023-01-11 21:56:30 33290:33290 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1250102Z STAGE:2023-01-11 21:56:30 33291:33291 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1251270Z STAGE:2023-01-11 21:56:30 33291:33291 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1252488Z STAGE:2023-01-11 21:56:30 33291:33291 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1253164Z ok (6.586s)
2023-01-11T22:18:07.1253442Z 
2023-01-11T22:18:07.1253975Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1254807Z Ran 1 test in 6.586s
2023-01-11T22:18:07.1255245Z 
2023-01-11T22:18:07.1255405Z OK
2023-01-11T22:18:07.1255670Z 
2023-01-11T22:18:07.1255916Z Generating XML reports...
2023-01-11T22:18:07.1256855Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215624.xml
2023-01-11T22:18:07.1257626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1257965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1258767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1259141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1259173Z 
2023-01-11T22:18:07.1259350Z Running tests...
2023-01-11T22:18:07.1259873Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1260531Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1261063Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1261515Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33409
2023-01-11T22:18:07.1261961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33410
2023-01-11T22:18:07.1262724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1263067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1263839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1264211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1264994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1265351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1266129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1266507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1266987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1267459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1268293Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1269078Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1269559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1270017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1270709Z STAGE:2023-01-11 21:56:38 33410:33410 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1271389Z STAGE:2023-01-11 21:56:38 33409:33409 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1272420Z STAGE:2023-01-11 21:56:39 33409:33409 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1273129Z STAGE:2023-01-11 21:56:39 33410:33410 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1273851Z STAGE:2023-01-11 21:56:39 33409:33409 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1274569Z STAGE:2023-01-11 21:56:39 33410:33410 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1275528Z STAGE:2023-01-11 21:56:39 33410:33410 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1276220Z STAGE:2023-01-11 21:56:39 33409:33409 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1276898Z STAGE:2023-01-11 21:56:39 33410:33410 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1277615Z STAGE:2023-01-11 21:56:39 33410:33410 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1278298Z STAGE:2023-01-11 21:56:39 33409:33409 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1278996Z STAGE:2023-01-11 21:56:39 33409:33409 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1279197Z ok (6.258s)
2023-01-11T22:18:07.1279230Z 
2023-01-11T22:18:07.1279756Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1279937Z Ran 1 test in 6.258s
2023-01-11T22:18:07.1280003Z 
2023-01-11T22:18:07.1280147Z OK
2023-01-11T22:18:07.1280173Z 
2023-01-11T22:18:07.1280418Z Generating XML reports...
2023-01-11T22:18:07.1281363Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215633.xml
2023-01-11T22:18:07.1282115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1282457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1283244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1283641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1283673Z 
2023-01-11T22:18:07.1283885Z Running tests...
2023-01-11T22:18:07.1284383Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1285017Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1285566Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.1285598Z 
2023-01-11T22:18:07.1286118Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1286331Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1286360Z 
2023-01-11T22:18:07.1286553Z OK (skipped=1)
2023-01-11T22:18:07.1286581Z 
2023-01-11T22:18:07.1286812Z Generating XML reports...
2023-01-11T22:18:07.1287730Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215642.xml
2023-01-11T22:18:07.1288501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1288814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1289601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1289996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1290027Z 
2023-01-11T22:18:07.1290238Z Running tests...
2023-01-11T22:18:07.1290765Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1291403Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1291928Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.1291958Z 
2023-01-11T22:18:07.1292484Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1292697Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1292726Z 
2023-01-11T22:18:07.1292894Z OK (skipped=1)
2023-01-11T22:18:07.1292921Z 
2023-01-11T22:18:07.1293152Z Generating XML reports...
2023-01-11T22:18:07.1294077Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215644.xml
2023-01-11T22:18:07.1295082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1295443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1296233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1296609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1296640Z 
2023-01-11T22:18:07.1296847Z Running tests...
2023-01-11T22:18:07.1297350Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1297997Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1298549Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1298984Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33594
2023-01-11T22:18:07.1299420Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33595
2023-01-11T22:18:07.1300192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1300531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1301314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1301696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1302414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1302764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1303543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1303932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1304423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1304906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1305752Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1306566Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1307031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1307463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1308150Z STAGE:2023-01-11 21:56:52 33594:33594 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1308824Z STAGE:2023-01-11 21:56:52 33595:33595 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1309510Z STAGE:2023-01-11 21:56:53 33594:33594 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1310200Z STAGE:2023-01-11 21:56:53 33595:33595 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1310906Z STAGE:2023-01-11 21:56:53 33594:33594 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1311602Z STAGE:2023-01-11 21:56:53 33595:33595 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1311791Z ok (6.596s)
2023-01-11T22:18:07.1311822Z 
2023-01-11T22:18:07.1312676Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1312902Z Ran 1 test in 6.597s
2023-01-11T22:18:07.1312933Z 
2023-01-11T22:18:07.1313110Z OK
2023-01-11T22:18:07.1313138Z 
2023-01-11T22:18:07.1313561Z Generating XML reports...
2023-01-11T22:18:07.1314621Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215647.xml
2023-01-11T22:18:07.1315402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1315738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1316528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1316904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1316938Z 
2023-01-11T22:18:07.1317122Z Running tests...
2023-01-11T22:18:07.1317652Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1318294Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1318861Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1319305Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33713
2023-01-11T22:18:07.1319736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33714
2023-01-11T22:18:07.1320513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1320859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1321606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1321985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1322741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1323088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1323877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1324259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1324746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1325214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1326062Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1326862Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1327334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1327804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1328491Z STAGE:2023-01-11 21:57:01 33713:33713 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1329171Z STAGE:2023-01-11 21:57:01 33714:33714 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1329875Z STAGE:2023-01-11 21:57:02 33713:33713 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1330553Z STAGE:2023-01-11 21:57:02 33714:33714 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1331330Z STAGE:2023-01-11 21:57:02 33713:33713 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1332042Z STAGE:2023-01-11 21:57:02 33714:33714 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1332213Z ok (6.586s)
2023-01-11T22:18:07.1332244Z 
2023-01-11T22:18:07.1332764Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1333133Z Ran 1 test in 6.586s
2023-01-11T22:18:07.1333240Z 
2023-01-11T22:18:07.1333425Z OK
2023-01-11T22:18:07.1333451Z 
2023-01-11T22:18:07.1333699Z Generating XML reports...
2023-01-11T22:18:07.1334629Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215656.xml
2023-01-11T22:18:07.1335376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1335732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1336491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1336880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1336909Z 
2023-01-11T22:18:07.1337114Z Running tests...
2023-01-11T22:18:07.1337637Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1338306Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1338823Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1340358Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81029 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.700s)
2023-01-11T22:18:07.1340391Z 
2023-01-11T22:18:07.1340918Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1341138Z Ran 1 test in 1.700s
2023-01-11T22:18:07.1341167Z 
2023-01-11T22:18:07.1341366Z OK (skipped=1)
2023-01-11T22:18:07.1341392Z 
2023-01-11T22:18:07.1341609Z Generating XML reports...
2023-01-11T22:18:07.1342521Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215705.xml
2023-01-11T22:18:07.1343299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1343643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1344425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1344801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1344833Z 
2023-01-11T22:18:07.1345040Z Running tests...
2023-01-11T22:18:07.1345570Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1346215Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1346729Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1348298Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81023 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.676s)
2023-01-11T22:18:07.1348335Z 
2023-01-11T22:18:07.1348861Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1349081Z Ran 1 test in 1.676s
2023-01-11T22:18:07.1349111Z 
2023-01-11T22:18:07.1349302Z OK (skipped=1)
2023-01-11T22:18:07.1349331Z 
2023-01-11T22:18:07.1349542Z Generating XML reports...
2023-01-11T22:18:07.1350445Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215709.xml
2023-01-11T22:18:07.1351213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1351549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1352972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1353374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1353405Z 
2023-01-11T22:18:07.1353607Z Running tests...
2023-01-11T22:18:07.1354148Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1354758Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1355300Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1355739Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33900
2023-01-11T22:18:07.1356163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33901
2023-01-11T22:18:07.1356927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1357301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1358100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1358485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1359242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1359550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1360325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1360695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1361185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1361690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1362501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1363325Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1363781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1364226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1364402Z ok (5.683s)
2023-01-11T22:18:07.1364433Z 
2023-01-11T22:18:07.1364968Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1365178Z Ran 1 test in 5.683s
2023-01-11T22:18:07.1365210Z 
2023-01-11T22:18:07.1365375Z OK
2023-01-11T22:18:07.1365404Z 
2023-01-11T22:18:07.1365661Z Generating XML reports...
2023-01-11T22:18:07.1366616Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215714.xml
2023-01-11T22:18:07.1367373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1367708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1368435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1368818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1368855Z 
2023-01-11T22:18:07.1369054Z Running tests...
2023-01-11T22:18:07.1369570Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1370206Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1370761Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1371478Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34015
2023-01-11T22:18:07.1371924Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34016
2023-01-11T22:18:07.1372720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1373038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1373794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1374183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1374929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1375269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1376053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1376439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1376934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1377387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1378221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1379028Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1379492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1379944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1380445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:18:07.1380921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:18:07.1381737Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.1382553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.1383005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:18:07.1383477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:18:07.1384284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:18:07.1385114Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:18:07.1385607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T22:18:07.1386100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T22:18:07.1386914Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2023-01-11T22:18:07.1387723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2023-01-11T22:18:07.1387926Z ok (6.550s)
2023-01-11T22:18:07.1387958Z 
2023-01-11T22:18:07.1388466Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1388657Z Ran 1 test in 6.551s
2023-01-11T22:18:07.1388686Z 
2023-01-11T22:18:07.1388858Z OK
2023-01-11T22:18:07.1388887Z 
2023-01-11T22:18:07.1389127Z Generating XML reports...
2023-01-11T22:18:07.1390255Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215722.xml
2023-01-11T22:18:07.1391006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1391347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1392467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1392872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1392906Z 
2023-01-11T22:18:07.1393090Z Running tests...
2023-01-11T22:18:07.1393647Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1394291Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1394792Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1395246Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34166
2023-01-11T22:18:07.1395686Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34167
2023-01-11T22:18:07.1396449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1396794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1397577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1397935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1398694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1399030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1399811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1400187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1400659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1401130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1401926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1402718Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1403188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1403642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1403841Z ok (5.802s)
2023-01-11T22:18:07.1403879Z 
2023-01-11T22:18:07.1404423Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1404633Z Ran 1 test in 5.802s
2023-01-11T22:18:07.1404662Z 
2023-01-11T22:18:07.1404829Z OK
2023-01-11T22:18:07.1404856Z 
2023-01-11T22:18:07.1405087Z Generating XML reports...
2023-01-11T22:18:07.1406019Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215731.xml
2023-01-11T22:18:07.1406757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1407106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1407879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1408252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1408453Z 
2023-01-11T22:18:07.1408667Z Running tests...
2023-01-11T22:18:07.1409332Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1409996Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1410857Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:18:07.1410889Z 
2023-01-11T22:18:07.1411424Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1411617Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1411646Z 
2023-01-11T22:18:07.1411843Z OK (skipped=1)
2023-01-11T22:18:07.1411870Z 
2023-01-11T22:18:07.1412101Z Generating XML reports...
2023-01-11T22:18:07.1413040Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215739.xml
2023-01-11T22:18:07.1413807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1414163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1414927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1415307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1415336Z 
2023-01-11T22:18:07.1415527Z Running tests...
2023-01-11T22:18:07.1416060Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1416687Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1417544Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:18:07.1417575Z 
2023-01-11T22:18:07.1418102Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1418322Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1418356Z 
2023-01-11T22:18:07.1418559Z OK (skipped=1)
2023-01-11T22:18:07.1418586Z 
2023-01-11T22:18:07.1418823Z Generating XML reports...
2023-01-11T22:18:07.1419745Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215742.xml
2023-01-11T22:18:07.1420505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1420850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1421611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1421998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1422028Z 
2023-01-11T22:18:07.1422237Z Running tests...
2023-01-11T22:18:07.1422755Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1423445Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1424332Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:18:07.1424367Z 
2023-01-11T22:18:07.1424877Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1425057Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1425085Z 
2023-01-11T22:18:07.1425278Z OK (skipped=1)
2023-01-11T22:18:07.1425305Z 
2023-01-11T22:18:07.1425538Z Generating XML reports...
2023-01-11T22:18:07.1426466Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215744.xml
2023-01-11T22:18:07.1427248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1427596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1428588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1428980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1429010Z 
2023-01-11T22:18:07.1429217Z Running tests...
2023-01-11T22:18:07.1429715Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1430361Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1431279Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:18:07.1431312Z 
2023-01-11T22:18:07.1432155Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1432772Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1432811Z 
2023-01-11T22:18:07.1432983Z OK (skipped=1)
2023-01-11T22:18:07.1433004Z 
2023-01-11T22:18:07.1433151Z Generating XML reports...
2023-01-11T22:18:07.1433692Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215746.xml
2023-01-11T22:18:07.1434080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1434241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1434625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1434816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1434836Z 
2023-01-11T22:18:07.1434944Z Running tests...
2023-01-11T22:18:07.1435213Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1435526Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1435940Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:18:07.1435964Z 
2023-01-11T22:18:07.1436226Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1436321Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1436359Z 
2023-01-11T22:18:07.1436449Z OK (skipped=1)
2023-01-11T22:18:07.1436468Z 
2023-01-11T22:18:07.1436591Z Generating XML reports...
2023-01-11T22:18:07.1437040Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215749.xml
2023-01-11T22:18:07.1437408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1437582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1437956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1438150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1438171Z 
2023-01-11T22:18:07.1438280Z Running tests...
2023-01-11T22:18:07.1438526Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1438834Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1439242Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:18:07.1439262Z 
2023-01-11T22:18:07.1439522Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1439633Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1439653Z 
2023-01-11T22:18:07.1439762Z OK (skipped=1)
2023-01-11T22:18:07.1439780Z 
2023-01-11T22:18:07.1439907Z Generating XML reports...
2023-01-11T22:18:07.1440351Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215751.xml
2023-01-11T22:18:07.1440978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1441145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1441525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1441715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1441734Z 
2023-01-11T22:18:07.1441842Z Running tests...
2023-01-11T22:18:07.1442101Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1442409Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1442826Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:18:07.1442846Z 
2023-01-11T22:18:07.1443109Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1443222Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1443241Z 
2023-01-11T22:18:07.1443329Z OK (skipped=1)
2023-01-11T22:18:07.1443347Z 
2023-01-11T22:18:07.1443469Z Generating XML reports...
2023-01-11T22:18:07.1443913Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215753.xml
2023-01-11T22:18:07.1444277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1444449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1444827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1445016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1445036Z 
2023-01-11T22:18:07.1445142Z Running tests...
2023-01-11T22:18:07.1445389Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1445701Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1446108Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:18:07.1446128Z 
2023-01-11T22:18:07.1446384Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1446494Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1446513Z 
2023-01-11T22:18:07.1446618Z OK (skipped=1)
2023-01-11T22:18:07.1446637Z 
2023-01-11T22:18:07.1446759Z Generating XML reports...
2023-01-11T22:18:07.1447197Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215756.xml
2023-01-11T22:18:07.1447561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1447723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1448099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1448284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1448304Z 
2023-01-11T22:18:07.1448409Z Running tests...
2023-01-11T22:18:07.1448667Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1448974Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1449368Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:18:07.1449387Z 
2023-01-11T22:18:07.1449645Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1449758Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1449777Z 
2023-01-11T22:18:07.1449930Z OK (skipped=1)
2023-01-11T22:18:07.1449949Z 
2023-01-11T22:18:07.1450112Z Generating XML reports...
2023-01-11T22:18:07.1450570Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215758.xml
2023-01-11T22:18:07.1450937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1451111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1451486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1451676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1451696Z 
2023-01-11T22:18:07.1451803Z Running tests...
2023-01-11T22:18:07.1452060Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1452351Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1452663Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.1452683Z 
2023-01-11T22:18:07.1482268Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1482423Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1482446Z 
2023-01-11T22:18:07.1482564Z OK (skipped=1)
2023-01-11T22:18:07.1482583Z 
2023-01-11T22:18:07.1482718Z Generating XML reports...
2023-01-11T22:18:07.1483218Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215801.xml
2023-01-11T22:18:07.1483593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1483768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1484152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1484353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1484373Z 
2023-01-11T22:18:07.1484480Z Running tests...
2023-01-11T22:18:07.1484730Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1485044Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1485442Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:18:07.1485462Z 
2023-01-11T22:18:07.1485723Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1485833Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1485852Z 
2023-01-11T22:18:07.1485957Z OK (skipped=1)
2023-01-11T22:18:07.1485975Z 
2023-01-11T22:18:07.1486096Z Generating XML reports...
2023-01-11T22:18:07.1486540Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215803.xml
2023-01-11T22:18:07.1486914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1487071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1487450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1487638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1487658Z 
2023-01-11T22:18:07.1487764Z Running tests...
2023-01-11T22:18:07.1488025Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1488335Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1488742Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:18:07.1488902Z 
2023-01-11T22:18:07.1489238Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1489337Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1489375Z 
2023-01-11T22:18:07.1489463Z OK (skipped=1)
2023-01-11T22:18:07.1489481Z 
2023-01-11T22:18:07.1489604Z Generating XML reports...
2023-01-11T22:18:07.1490051Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215805.xml
2023-01-11T22:18:07.1490418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1490589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1490963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1491151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1491171Z 
2023-01-11T22:18:07.1491283Z Running tests...
2023-01-11T22:18:07.1491527Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1491837Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1492231Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:18:07.1492251Z 
2023-01-11T22:18:07.1492507Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1492615Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1492634Z 
2023-01-11T22:18:07.1492739Z OK (skipped=1)
2023-01-11T22:18:07.1492757Z 
2023-01-11T22:18:07.1492879Z Generating XML reports...
2023-01-11T22:18:07.1493321Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215808.xml
2023-01-11T22:18:07.1493687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1493850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1494221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1494409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1494428Z 
2023-01-11T22:18:07.1494534Z Running tests...
2023-01-11T22:18:07.1494794Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1495102Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1495388Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.1495408Z 
2023-01-11T22:18:07.1495663Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1495771Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1495794Z 
2023-01-11T22:18:07.1495883Z OK (skipped=1)
2023-01-11T22:18:07.1495904Z 
2023-01-11T22:18:07.1496025Z Generating XML reports...
2023-01-11T22:18:07.1496462Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215810.xml
2023-01-11T22:18:07.1496826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1496997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1497369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1497555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1497574Z 
2023-01-11T22:18:07.1497680Z Running tests...
2023-01-11T22:18:07.1497920Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1498227Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1498598Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.1498620Z 
2023-01-11T22:18:07.1498882Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1498992Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1499012Z 
2023-01-11T22:18:07.1499116Z OK (skipped=1)
2023-01-11T22:18:07.1499134Z 
2023-01-11T22:18:07.1499255Z Generating XML reports...
2023-01-11T22:18:07.1499697Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215812.xml
2023-01-11T22:18:07.1500059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1500214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1500596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1500785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1500804Z 
2023-01-11T22:18:07.1500908Z Running tests...
2023-01-11T22:18:07.1501168Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1501479Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1501751Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.1501771Z 
2023-01-11T22:18:07.1502026Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1502135Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1502154Z 
2023-01-11T22:18:07.1502240Z OK (skipped=1)
2023-01-11T22:18:07.1502259Z 
2023-01-11T22:18:07.1502378Z Generating XML reports...
2023-01-11T22:18:07.1502823Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215815.xml
2023-01-11T22:18:07.1503193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1503367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1503741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1503930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1503949Z 
2023-01-11T22:18:07.1504054Z Running tests...
2023-01-11T22:18:07.1504293Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1504602Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1504877Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.1504900Z 
2023-01-11T22:18:07.1505158Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1505267Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1505287Z 
2023-01-11T22:18:07.1505393Z OK (skipped=1)
2023-01-11T22:18:07.1505412Z 
2023-01-11T22:18:07.1505532Z Generating XML reports...
2023-01-11T22:18:07.1505972Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215817.xml
2023-01-11T22:18:07.1506338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1506492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1506866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1507053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1507126Z 
2023-01-11T22:18:07.1507238Z Running tests...
2023-01-11T22:18:07.1507538Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1507855Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1508128Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.1508148Z 
2023-01-11T22:18:07.1508405Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1508514Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1508536Z 
2023-01-11T22:18:07.1508625Z OK (skipped=1)
2023-01-11T22:18:07.1508644Z 
2023-01-11T22:18:07.1508766Z Generating XML reports...
2023-01-11T22:18:07.1509210Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215820.xml
2023-01-11T22:18:07.1509576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1509758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1510131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1510320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1510339Z 
2023-01-11T22:18:07.1510445Z Running tests...
2023-01-11T22:18:07.1510702Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1510994Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1511256Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.1511276Z 
2023-01-11T22:18:07.1511534Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1511641Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1511663Z 
2023-01-11T22:18:07.1511767Z OK (skipped=1)
2023-01-11T22:18:07.1511789Z 
2023-01-11T22:18:07.1512175Z Generating XML reports...
2023-01-11T22:18:07.1512635Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215822.xml
2023-01-11T22:18:07.1512996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1513167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1513519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1513706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1513725Z 
2023-01-11T22:18:07.1513831Z Running tests...
2023-01-11T22:18:07.1514089Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1514393Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1514666Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.1514686Z 
2023-01-11T22:18:07.1514941Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1515048Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1515067Z 
2023-01-11T22:18:07.1515154Z OK (skipped=1)
2023-01-11T22:18:07.1515190Z 
2023-01-11T22:18:07.1515293Z Generating XML reports...
2023-01-11T22:18:07.1515731Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215824.xml
2023-01-11T22:18:07.1516093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1516266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1516636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1516976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1516997Z 
2023-01-11T22:18:07.1517107Z Running tests...
2023-01-11T22:18:07.1517368Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1517659Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1517930Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.1517949Z 
2023-01-11T22:18:07.1518204Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1518312Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1518332Z 
2023-01-11T22:18:07.1518435Z OK (skipped=1)
2023-01-11T22:18:07.1518454Z 
2023-01-11T22:18:07.1518573Z Generating XML reports...
2023-01-11T22:18:07.1519012Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215827.xml
2023-01-11T22:18:07.1519387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1519559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1519917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1520106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1520125Z 
2023-01-11T22:18:07.1520230Z Running tests...
2023-01-11T22:18:07.1520489Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1520794Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1521055Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.1521078Z 
2023-01-11T22:18:07.1521337Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1521446Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1521465Z 
2023-01-11T22:18:07.1521553Z OK (skipped=1)
2023-01-11T22:18:07.1521589Z 
2023-01-11T22:18:07.1521693Z Generating XML reports...
2023-01-11T22:18:07.1522129Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215829.xml
2023-01-11T22:18:07.1522493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1522665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1523034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1523221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1523240Z 
2023-01-11T22:18:07.1523350Z Running tests...
2023-01-11T22:18:07.1523610Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1523902Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1524160Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.1524179Z 
2023-01-11T22:18:07.1524434Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1524542Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1524561Z 
2023-01-11T22:18:07.1524666Z OK (skipped=1)
2023-01-11T22:18:07.1524685Z 
2023-01-11T22:18:07.1524805Z Generating XML reports...
2023-01-11T22:18:07.1525241Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215832.xml
2023-01-11T22:18:07.1525602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1525886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1526250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1526439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1526458Z 
2023-01-11T22:18:07.1526563Z Running tests...
2023-01-11T22:18:07.1526823Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1527130Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1527388Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.1527407Z 
2023-01-11T22:18:07.1527662Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1527769Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1527788Z 
2023-01-11T22:18:07.1527892Z OK (skipped=1)
2023-01-11T22:18:07.1527916Z 
2023-01-11T22:18:07.1528022Z Generating XML reports...
2023-01-11T22:18:07.1528464Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215834.xml
2023-01-11T22:18:07.1528828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1528999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1529373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1529562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1529581Z 
2023-01-11T22:18:07.1529687Z Running tests...
2023-01-11T22:18:07.1529945Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1530237Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1530567Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2023-01-11T22:18:07.1530588Z 
2023-01-11T22:18:07.1530845Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1530953Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1530971Z 
2023-01-11T22:18:07.1531076Z OK (skipped=1)
2023-01-11T22:18:07.1531094Z 
2023-01-11T22:18:07.1531216Z Generating XML reports...
2023-01-11T22:18:07.1531654Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215836.xml
2023-01-11T22:18:07.1532019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1532190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1532545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1532740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1532759Z 
2023-01-11T22:18:07.1532863Z Running tests...
2023-01-11T22:18:07.1533120Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1533426Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1533711Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: CUDA all_reduce multigpu skipped for NCCL (0.002s)
2023-01-11T22:18:07.1533730Z 
2023-01-11T22:18:07.1533984Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1534093Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1534112Z 
2023-01-11T22:18:07.1534217Z OK (skipped=1)
2023-01-11T22:18:07.1534236Z 
2023-01-11T22:18:07.1534339Z Generating XML reports...
2023-01-11T22:18:07.1534778Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215839.xml
2023-01-11T22:18:07.1535242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1535419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1535792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1535978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1535998Z 
2023-01-11T22:18:07.1536103Z Running tests...
2023-01-11T22:18:07.1536357Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1536646Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1536910Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.1536930Z 
2023-01-11T22:18:07.1537193Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1537301Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1537320Z 
2023-01-11T22:18:07.1537423Z OK (skipped=1)
2023-01-11T22:18:07.1537442Z 
2023-01-11T22:18:07.1537563Z Generating XML reports...
2023-01-11T22:18:07.1537998Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215841.xml
2023-01-11T22:18:07.1538360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1538530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1538884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1539072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1539090Z 
2023-01-11T22:18:07.1539196Z Running tests...
2023-01-11T22:18:07.1539461Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1539771Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1540036Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1540255Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35172
2023-01-11T22:18:07.1540470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35173
2023-01-11T22:18:07.1540838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1540993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1541367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1541554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1541920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1542091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1542458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1542645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1542888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1543112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1543507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1543897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1544220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1544448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1544548Z ok (5.814s)
2023-01-11T22:18:07.1544567Z 
2023-01-11T22:18:07.1544832Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1544942Z Ran 1 test in 5.814s
2023-01-11T22:18:07.1544962Z 
2023-01-11T22:18:07.1545052Z OK
2023-01-11T22:18:07.1545072Z 
2023-01-11T22:18:07.1545176Z Generating XML reports...
2023-01-11T22:18:07.1545621Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215844.xml
2023-01-11T22:18:07.1545987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1546158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1546538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1546727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1546746Z 
2023-01-11T22:18:07.1546852Z Running tests...
2023-01-11T22:18:07.1547107Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1547399Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1547657Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.1547676Z 
2023-01-11T22:18:07.1547928Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1548036Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1548055Z 
2023-01-11T22:18:07.1548160Z OK (skipped=1)
2023-01-11T22:18:07.1548179Z 
2023-01-11T22:18:07.1548305Z Generating XML reports...
2023-01-11T22:18:07.1548750Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215852.xml
2023-01-11T22:18:07.1549115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1549286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1549643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1549835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1549855Z 
2023-01-11T22:18:07.1549959Z Running tests...
2023-01-11T22:18:07.1550217Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1550523Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1550789Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.1550812Z 
2023-01-11T22:18:07.1551068Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1551177Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1551195Z 
2023-01-11T22:18:07.1551300Z OK (skipped=1)
2023-01-11T22:18:07.1551319Z 
2023-01-11T22:18:07.1551421Z Generating XML reports...
2023-01-11T22:18:07.1552102Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215854.xml
2023-01-11T22:18:07.1552493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1552667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1553040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1553228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1553378Z 
2023-01-11T22:18:07.1553494Z Running tests...
2023-01-11T22:18:07.1553757Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1554068Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1554317Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.1554336Z 
2023-01-11T22:18:07.1554591Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1554699Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1554719Z 
2023-01-11T22:18:07.1554823Z OK (skipped=1)
2023-01-11T22:18:07.1554842Z 
2023-01-11T22:18:07.1554961Z Generating XML reports...
2023-01-11T22:18:07.1555402Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215857.xml
2023-01-11T22:18:07.1555773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1555945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1556302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1556492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1556511Z 
2023-01-11T22:18:07.1556616Z Running tests...
2023-01-11T22:18:07.1556873Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1557184Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1557442Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1557658Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35386
2023-01-11T22:18:07.1557878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35387
2023-01-11T22:18:07.1558241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1558395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1558766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1558954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1559313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1559482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1559847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1560030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1560280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1560503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1560898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1561289Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1561514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1561739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1562073Z STAGE:2023-01-11 21:59:04 35386:35386 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1562394Z STAGE:2023-01-11 21:59:04 35387:35387 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1562826Z STAGE:2023-01-11 21:59:04 35387:35387 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1563163Z STAGE:2023-01-11 21:59:04 35386:35386 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1563493Z STAGE:2023-01-11 21:59:04 35387:35387 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1563836Z STAGE:2023-01-11 21:59:04 35386:35386 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1564161Z STAGE:2023-01-11 21:59:04 35386:35386 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1564479Z STAGE:2023-01-11 21:59:04 35387:35387 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1564811Z STAGE:2023-01-11 21:59:04 35386:35386 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1565149Z STAGE:2023-01-11 21:59:04 35386:35386 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1565475Z STAGE:2023-01-11 21:59:04 35387:35387 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1565812Z STAGE:2023-01-11 21:59:04 35387:35387 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1566124Z STAGE:2023-01-11 21:59:04 35386:35386 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1566429Z STAGE:2023-01-11 21:59:04 35387:35387 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1566750Z STAGE:2023-01-11 21:59:04 35386:35386 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1567085Z STAGE:2023-01-11 21:59:04 35386:35386 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1567408Z STAGE:2023-01-11 21:59:04 35387:35387 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1567749Z STAGE:2023-01-11 21:59:04 35387:35387 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1567849Z ok (5.705s)
2023-01-11T22:18:07.1567870Z 
2023-01-11T22:18:07.1568125Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1568230Z Ran 1 test in 5.706s
2023-01-11T22:18:07.1568250Z 
2023-01-11T22:18:07.1568333Z OK
2023-01-11T22:18:07.1568353Z 
2023-01-11T22:18:07.1568456Z Generating XML reports...
2023-01-11T22:18:07.1568898Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215859.xml
2023-01-11T22:18:07.1569264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1569433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1569806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1569998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1570018Z 
2023-01-11T22:18:07.1570122Z Running tests...
2023-01-11T22:18:07.1570376Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1570680Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1570931Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1571142Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35504
2023-01-11T22:18:07.1571345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35505
2023-01-11T22:18:07.1571707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1571876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1572343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1572530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1572888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1573040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1573404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1573586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1573821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1574057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1574450Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1574840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1575062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1575281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1575595Z STAGE:2023-01-11 21:59:12 35504:35504 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1575904Z STAGE:2023-01-11 21:59:12 35505:35505 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1576229Z STAGE:2023-01-11 21:59:12 35505:35505 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1576549Z STAGE:2023-01-11 21:59:12 35504:35504 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1577116Z STAGE:2023-01-11 21:59:12 35504:35504 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:59:12 35505:35505 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1577137Z 
2023-01-11T22:18:07.1577451Z STAGE:2023-01-11 21:59:13 35504:35504 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1577756Z STAGE:2023-01-11 21:59:13 35505:35505 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1578072Z STAGE:2023-01-11 21:59:13 35504:35504 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1578402Z STAGE:2023-01-11 21:59:13 35504:35504 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1578716Z STAGE:2023-01-11 21:59:13 35505:35505 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1579038Z STAGE:2023-01-11 21:59:13 35505:35505 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1579355Z STAGE:2023-01-11 21:59:13 35504:35504 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1579667Z STAGE:2023-01-11 21:59:13 35505:35505 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1579984Z STAGE:2023-01-11 21:59:13 35504:35504 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1580311Z STAGE:2023-01-11 21:59:13 35504:35504 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1580625Z STAGE:2023-01-11 21:59:13 35505:35505 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1580954Z STAGE:2023-01-11 21:59:13 35505:35505 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1581044Z ok (5.670s)
2023-01-11T22:18:07.1581064Z 
2023-01-11T22:18:07.1581315Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1581464Z Ran 1 test in 5.670s
2023-01-11T22:18:07.1581483Z 
2023-01-11T22:18:07.1581603Z OK
2023-01-11T22:18:07.1581623Z 
2023-01-11T22:18:07.1581738Z Generating XML reports...
2023-01-11T22:18:07.1582176Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215907.xml
2023-01-11T22:18:07.1582529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1582690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1583054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1583232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1583252Z 
2023-01-11T22:18:07.1583340Z Running tests...
2023-01-11T22:18:07.1583590Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1583897Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1584159Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1584363Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35622
2023-01-11T22:18:07.1584565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35623
2023-01-11T22:18:07.1584918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1585077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1585434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1585611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1585957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1586124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1586481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1586657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1586892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1587126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1587517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1587892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1588116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1588342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1588670Z STAGE:2023-01-11 21:59:20 35623:35623 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1588982Z STAGE:2023-01-11 21:59:20 35622:35622 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1589304Z STAGE:2023-01-11 21:59:21 35622:35622 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1589628Z STAGE:2023-01-11 21:59:21 35623:35623 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1589964Z STAGE:2023-01-11 21:59:21 35622:35622 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1590302Z STAGE:2023-01-11 21:59:21 35623:35623 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1590652Z STAGE:2023-01-11 21:59:21 35622:35622 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1591031Z STAGE:2023-01-11 21:59:21 35623:35623 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1591356Z STAGE:2023-01-11 21:59:21 35622:35622 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1591689Z STAGE:2023-01-11 21:59:21 35622:35622 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1592247Z STAGE:2023-01-11 21:59:21 35623:35623 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1592595Z STAGE:2023-01-11 21:59:21 35623:35623 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1592910Z STAGE:2023-01-11 21:59:21 35622:35622 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1593292Z STAGE:2023-01-11 21:59:21 35623:35623 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1593625Z STAGE:2023-01-11 21:59:21 35622:35622 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1593934Z STAGE:2023-01-11 21:59:21 35623:35623 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1594262Z STAGE:2023-01-11 21:59:21 35622:35622 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1594590Z STAGE:2023-01-11 21:59:21 35623:35623 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1594680Z ok (5.682s)
2023-01-11T22:18:07.1594702Z 
2023-01-11T22:18:07.1594952Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1595053Z Ran 1 test in 5.682s
2023-01-11T22:18:07.1595073Z 
2023-01-11T22:18:07.1595159Z OK
2023-01-11T22:18:07.1595178Z 
2023-01-11T22:18:07.1595294Z Generating XML reports...
2023-01-11T22:18:07.1595740Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215916.xml
2023-01-11T22:18:07.1596102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1596275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1596648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1596830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1596849Z 
2023-01-11T22:18:07.1596944Z Running tests...
2023-01-11T22:18:07.1597192Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1597488Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1597718Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2023-01-11T22:18:07.1597738Z 
2023-01-11T22:18:07.1597979Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1598086Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1598106Z 
2023-01-11T22:18:07.1598201Z OK (skipped=1)
2023-01-11T22:18:07.1598220Z 
2023-01-11T22:18:07.1598330Z Generating XML reports...
2023-01-11T22:18:07.1598759Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215924.xml
2023-01-11T22:18:07.1599109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1599269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1599628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1599804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1599824Z 
2023-01-11T22:18:07.1599913Z Running tests...
2023-01-11T22:18:07.1600156Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1600601Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1600851Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2023-01-11T22:18:07.1600871Z 
2023-01-11T22:18:07.1601122Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1601220Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1601240Z 
2023-01-11T22:18:07.1601332Z OK (skipped=1)
2023-01-11T22:18:07.1601351Z 
2023-01-11T22:18:07.1601461Z Generating XML reports...
2023-01-11T22:18:07.1601894Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215926.xml
2023-01-11T22:18:07.1602244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1602406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1602776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1602954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1602974Z 
2023-01-11T22:18:07.1603070Z Running tests...
2023-01-11T22:18:07.1603317Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1603612Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1603852Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1604051Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35806
2023-01-11T22:18:07.1604254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35807
2023-01-11T22:18:07.1604607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1604777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1605142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1605317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1605666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1605824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1606180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1606350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1606580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1606814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1607200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1607580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1607796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1608008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1608098Z ok (5.616s)
2023-01-11T22:18:07.1608118Z 
2023-01-11T22:18:07.1608362Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1608460Z Ran 1 test in 5.616s
2023-01-11T22:18:07.1608480Z 
2023-01-11T22:18:07.1608558Z OK
2023-01-11T22:18:07.1608577Z 
2023-01-11T22:18:07.1608687Z Generating XML reports...
2023-01-11T22:18:07.1609223Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215929.xml
2023-01-11T22:18:07.1609585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1609746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1610107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1610283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1610302Z 
2023-01-11T22:18:07.1610391Z Running tests...
2023-01-11T22:18:07.1610638Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1610935Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1611186Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1611397Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35923
2023-01-11T22:18:07.1611600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35924
2023-01-11T22:18:07.1611954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1612113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1612467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1612643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1612986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1613145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1613502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1613678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1613912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1614141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1614521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1614894Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1615111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1615323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1615419Z ok (5.656s)
2023-01-11T22:18:07.1615439Z 
2023-01-11T22:18:07.1615700Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1615808Z Ran 1 test in 5.656s
2023-01-11T22:18:07.1615827Z 
2023-01-11T22:18:07.1615910Z OK
2023-01-11T22:18:07.1615929Z 
2023-01-11T22:18:07.1616042Z Generating XML reports...
2023-01-11T22:18:07.1616480Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215937.xml
2023-01-11T22:18:07.1616831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1616999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1617361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1617537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1617619Z 
2023-01-11T22:18:07.1617721Z Running tests...
2023-01-11T22:18:07.1618012Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1618318Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1618564Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2023-01-11T22:18:07.1618584Z 
2023-01-11T22:18:07.1618825Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1618927Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1618946Z 
2023-01-11T22:18:07.1619044Z OK (skipped=1)
2023-01-11T22:18:07.1619062Z 
2023-01-11T22:18:07.1619175Z Generating XML reports...
2023-01-11T22:18:07.1619607Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215945.xml
2023-01-11T22:18:07.1619962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1620138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1620508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1620688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1620707Z 
2023-01-11T22:18:07.1620795Z Running tests...
2023-01-11T22:18:07.1621047Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1621345Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1621601Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1621807Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36073
2023-01-11T22:18:07.1622011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36074
2023-01-11T22:18:07.1622375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1622540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1622897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1623079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1623431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1623595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1623950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1624133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1624371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1624609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1624995Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1625368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1625589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1625815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:18:07.1626033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1626262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:18:07.1626749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.1627138Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.1627232Z ok (5.653s)
2023-01-11T22:18:07.1627252Z 
2023-01-11T22:18:07.1627506Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1627599Z Ran 1 test in 5.654s
2023-01-11T22:18:07.1627618Z 
2023-01-11T22:18:07.1627700Z OK
2023-01-11T22:18:07.1627719Z 
2023-01-11T22:18:07.1627833Z Generating XML reports...
2023-01-11T22:18:07.1628272Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215947.xml
2023-01-11T22:18:07.1628630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1628792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1629168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1629351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1629371Z 
2023-01-11T22:18:07.1629470Z Running tests...
2023-01-11T22:18:07.1629709Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1630011Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1630259Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2023-01-11T22:18:07.1630279Z 
2023-01-11T22:18:07.1630576Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1630679Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1630698Z 
2023-01-11T22:18:07.1630796Z OK (skipped=1)
2023-01-11T22:18:07.1630814Z 
2023-01-11T22:18:07.1630932Z Generating XML reports...
2023-01-11T22:18:07.1631376Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215955.xml
2023-01-11T22:18:07.1631727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1632114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1632506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1632691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1632712Z 
2023-01-11T22:18:07.1632817Z Running tests...
2023-01-11T22:18:07.1633070Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1633371Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1633630Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1633844Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36225
2023-01-11T22:18:07.1634041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36226
2023-01-11T22:18:07.1634402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1634568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1634933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1635116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1635465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1635629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1636131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1636306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1636543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1636780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1637175Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1637561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1637780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1637998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1638152Z skip: Skipped due to small world size. (4.250s)
2023-01-11T22:18:07.1638173Z 
2023-01-11T22:18:07.1638427Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1638521Z Ran 1 test in 4.250s
2023-01-11T22:18:07.1638554Z 
2023-01-11T22:18:07.1638642Z OK (skipped=1)
2023-01-11T22:18:07.1638661Z 
2023-01-11T22:18:07.1638781Z Generating XML reports...
2023-01-11T22:18:07.1639218Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215958.xml
2023-01-11T22:18:07.1639578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1639742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1640114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1640297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1640320Z 
2023-01-11T22:18:07.1640417Z Running tests...
2023-01-11T22:18:07.1640659Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1640967Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1641238Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T22:18:07.1641258Z 
2023-01-11T22:18:07.1641507Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1641614Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1641634Z 
2023-01-11T22:18:07.1641737Z OK (skipped=1)
2023-01-11T22:18:07.1641756Z 
2023-01-11T22:18:07.1641873Z Generating XML reports...
2023-01-11T22:18:07.1642310Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220005.xml
2023-01-11T22:18:07.1642682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1642841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1643209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1643386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1643406Z 
2023-01-11T22:18:07.1643506Z Running tests...
2023-01-11T22:18:07.1643762Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1644057Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1644340Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T22:18:07.1644360Z 
2023-01-11T22:18:07.1644611Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1644797Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1644830Z 
2023-01-11T22:18:07.1644921Z OK (skipped=1)
2023-01-11T22:18:07.1644939Z 
2023-01-11T22:18:07.1645053Z Generating XML reports...
2023-01-11T22:18:07.1645495Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220007.xml
2023-01-11T22:18:07.1645858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1646022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1646386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1646563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1646582Z 
2023-01-11T22:18:07.1646682Z Running tests...
2023-01-11T22:18:07.1646924Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1647232Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1647501Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1648247Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81030 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.686s)
2023-01-11T22:18:07.1648267Z 
2023-01-11T22:18:07.1648521Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1648622Z Ran 1 test in 1.686s
2023-01-11T22:18:07.1648641Z 
2023-01-11T22:18:07.1648742Z OK (skipped=1)
2023-01-11T22:18:07.1648761Z 
2023-01-11T22:18:07.1648874Z Generating XML reports...
2023-01-11T22:18:07.1649315Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220009.xml
2023-01-11T22:18:07.1649678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1649835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1650211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1650397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1650418Z 
2023-01-11T22:18:07.1650521Z Running tests...
2023-01-11T22:18:07.1650780Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1651086Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1651372Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1651590Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36430
2023-01-11T22:18:07.1651786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36431
2023-01-11T22:18:07.1652150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1652317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1652685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1652865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1653220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1653382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1653858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1654046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1654272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1654511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1654910Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1655300Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1655519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1655738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1656073Z STAGE:2023-01-11 22:00:18 36431:36431 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1656391Z STAGE:2023-01-11 22:00:19 36430:36430 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.1656714Z STAGE:2023-01-11 22:00:19 36430:36430 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1657026Z STAGE:2023-01-11 22:00:19 36431:36431 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.1657361Z STAGE:2023-01-11 22:00:19 36431:36431 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1657699Z STAGE:2023-01-11 22:00:19 36430:36430 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.1657797Z ok (6.394s)
2023-01-11T22:18:07.1657817Z 
2023-01-11T22:18:07.1658072Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1658175Z Ran 1 test in 6.394s
2023-01-11T22:18:07.1658197Z 
2023-01-11T22:18:07.1658281Z OK
2023-01-11T22:18:07.1658303Z 
2023-01-11T22:18:07.1658422Z Generating XML reports...
2023-01-11T22:18:07.1658849Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220014.xml
2023-01-11T22:18:07.1659207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1659372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1659740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1659923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1659942Z 
2023-01-11T22:18:07.1660041Z Running tests...
2023-01-11T22:18:07.1660297Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1660598Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1660891Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T22:18:07.1660911Z 
2023-01-11T22:18:07.1661152Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1661259Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1661278Z 
2023-01-11T22:18:07.1661382Z OK (skipped=1)
2023-01-11T22:18:07.1661400Z 
2023-01-11T22:18:07.1661517Z Generating XML reports...
2023-01-11T22:18:07.1661957Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220023.xml
2023-01-11T22:18:07.1662321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1662491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1662917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1663149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1663169Z 
2023-01-11T22:18:07.1663258Z Running tests...
2023-01-11T22:18:07.1663517Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1663825Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1664112Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1664851Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81004 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.665s)
2023-01-11T22:18:07.1664872Z 
2023-01-11T22:18:07.1665130Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1665233Z Ran 1 test in 1.665s
2023-01-11T22:18:07.1665253Z 
2023-01-11T22:18:07.1665355Z OK (skipped=1)
2023-01-11T22:18:07.1665373Z 
2023-01-11T22:18:07.1665490Z Generating XML reports...
2023-01-11T22:18:07.1665924Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220025.xml
2023-01-11T22:18:07.1666272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1666442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1666817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1667002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1667021Z 
2023-01-11T22:18:07.1667121Z Running tests...
2023-01-11T22:18:07.1667388Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1667695Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1667979Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T22:18:07.1667999Z 
2023-01-11T22:18:07.1668252Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1668344Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1668363Z 
2023-01-11T22:18:07.1668466Z OK (skipped=1)
2023-01-11T22:18:07.1668484Z 
2023-01-11T22:18:07.1668602Z Generating XML reports...
2023-01-11T22:18:07.1669040Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220029.xml
2023-01-11T22:18:07.1669403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1669578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1669950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1670131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1670150Z 
2023-01-11T22:18:07.1670238Z Running tests...
2023-01-11T22:18:07.1670491Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1670798Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1671079Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1671293Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36651
2023-01-11T22:18:07.1671509Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36652
2023-01-11T22:18:07.1672232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1672420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1672809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1672978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1673336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1673503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1673866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1674050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1674291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1674537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1674923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1675297Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1675516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1675734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1675883Z skip: Skipped due to small world size. (4.291s)
2023-01-11T22:18:07.1675904Z 
2023-01-11T22:18:07.1676157Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1676259Z Ran 1 test in 4.292s
2023-01-11T22:18:07.1676281Z 
2023-01-11T22:18:07.1676381Z OK (skipped=1)
2023-01-11T22:18:07.1676402Z 
2023-01-11T22:18:07.1676522Z Generating XML reports...
2023-01-11T22:18:07.1676965Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220032.xml
2023-01-11T22:18:07.1677316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1677480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1677847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1678033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1678052Z 
2023-01-11T22:18:07.1678149Z Running tests...
2023-01-11T22:18:07.1678400Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1678699Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1678986Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T22:18:07.1679006Z 
2023-01-11T22:18:07.1679260Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1679353Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1679372Z 
2023-01-11T22:18:07.1679471Z OK (skipped=1)
2023-01-11T22:18:07.1679489Z 
2023-01-11T22:18:07.1679605Z Generating XML reports...
2023-01-11T22:18:07.1680036Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220038.xml
2023-01-11T22:18:07.1680393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1680557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1680969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1681206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1681225Z 
2023-01-11T22:18:07.1681323Z Running tests...
2023-01-11T22:18:07.1681567Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1681870Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1682157Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T22:18:07.1682177Z 
2023-01-11T22:18:07.1682424Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1682526Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1682545Z 
2023-01-11T22:18:07.1682645Z OK (skipped=1)
2023-01-11T22:18:07.1682663Z 
2023-01-11T22:18:07.1682780Z Generating XML reports...
2023-01-11T22:18:07.1683218Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220041.xml
2023-01-11T22:18:07.1683582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1683736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1684111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1684288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1684307Z 
2023-01-11T22:18:07.1684406Z Running tests...
2023-01-11T22:18:07.1684658Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1684957Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1685228Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1685443Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36822
2023-01-11T22:18:07.1685642Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36823
2023-01-11T22:18:07.1685999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1686170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1686537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1686718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1687076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1687240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1687602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1687784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1688012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1688245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1688635Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1689021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1689237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1689451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1689599Z ok (5.703s)
2023-01-11T22:18:07.1689619Z 
2023-01-11T22:18:07.1689913Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1690012Z Ran 1 test in 5.704s
2023-01-11T22:18:07.1690040Z 
2023-01-11T22:18:07.1690113Z OK
2023-01-11T22:18:07.1690132Z 
2023-01-11T22:18:07.1690255Z Generating XML reports...
2023-01-11T22:18:07.1690694Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220043.xml
2023-01-11T22:18:07.1691051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1691215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1691588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1691766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1691785Z 
2023-01-11T22:18:07.1691888Z Running tests...
2023-01-11T22:18:07.1692134Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1692437Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1692720Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1692933Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36939
2023-01-11T22:18:07.1693134Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36940
2023-01-11T22:18:07.1693492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1693660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1694021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1694198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1694549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1694719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1695080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1695256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1695489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1695719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1696109Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1696500Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1696710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1696926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1697026Z ok (5.658s)
2023-01-11T22:18:07.1697047Z 
2023-01-11T22:18:07.1697296Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1697397Z Ran 1 test in 5.658s
2023-01-11T22:18:07.1697416Z 
2023-01-11T22:18:07.1697504Z OK
2023-01-11T22:18:07.1697522Z 
2023-01-11T22:18:07.1697642Z Generating XML reports...
2023-01-11T22:18:07.1698077Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220051.xml
2023-01-11T22:18:07.1698436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1698693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1699074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1699254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1699273Z 
2023-01-11T22:18:07.1699372Z Running tests...
2023-01-11T22:18:07.1699623Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1699923Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1700218Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T22:18:07.1700238Z 
2023-01-11T22:18:07.1700484Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1700577Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1700611Z 
2023-01-11T22:18:07.1700701Z OK (skipped=1)
2023-01-11T22:18:07.1700722Z 
2023-01-11T22:18:07.1700837Z Generating XML reports...
2023-01-11T22:18:07.1701273Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220059.xml
2023-01-11T22:18:07.1701632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1701802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1702168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1702348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1702367Z 
2023-01-11T22:18:07.1702472Z Running tests...
2023-01-11T22:18:07.1702712Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1703013Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1703305Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1703519Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37089
2023-01-11T22:18:07.1703723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37090
2023-01-11T22:18:07.1704079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1704241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1704607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1704788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1705131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1705302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1705663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1705844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1706075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1706308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1706692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1707084Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1707294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1707623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:18:07.1707842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1708072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:18:07.1708461Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.1708841Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.1708939Z ok (5.762s)
2023-01-11T22:18:07.1708959Z 
2023-01-11T22:18:07.1709212Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1709315Z Ran 1 test in 5.762s
2023-01-11T22:18:07.1709335Z 
2023-01-11T22:18:07.1709407Z OK
2023-01-11T22:18:07.1709437Z 
2023-01-11T22:18:07.1709545Z Generating XML reports...
2023-01-11T22:18:07.1709984Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220102.xml
2023-01-11T22:18:07.1710348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1710514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1710881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1711065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1711085Z 
2023-01-11T22:18:07.1711185Z Running tests...
2023-01-11T22:18:07.1711436Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1711730Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1712266Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T22:18:07.1712291Z 
2023-01-11T22:18:07.1712560Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1712662Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1712683Z 
2023-01-11T22:18:07.1712781Z OK (skipped=1)
2023-01-11T22:18:07.1712800Z 
2023-01-11T22:18:07.1712921Z Generating XML reports...
2023-01-11T22:18:07.1713355Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220110.xml
2023-01-11T22:18:07.1713712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1713877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1714234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1714420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1714439Z 
2023-01-11T22:18:07.1714535Z Running tests...
2023-01-11T22:18:07.1714786Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1715087Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1715371Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1715578Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37241
2023-01-11T22:18:07.1715786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37242
2023-01-11T22:18:07.1716131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1716302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1716822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1717009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1717359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1717518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1717883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1718066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1718303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1718528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1718925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1719312Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1719527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1719745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1719899Z skip: Skipped due to small world size. (4.287s)
2023-01-11T22:18:07.1719919Z 
2023-01-11T22:18:07.1720177Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1720281Z Ran 1 test in 4.288s
2023-01-11T22:18:07.1720301Z 
2023-01-11T22:18:07.1720397Z OK (skipped=1)
2023-01-11T22:18:07.1720417Z 
2023-01-11T22:18:07.1720521Z Generating XML reports...
2023-01-11T22:18:07.1720961Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220112.xml
2023-01-11T22:18:07.1721326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1721492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1721859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1722044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1722063Z 
2023-01-11T22:18:07.1722161Z Running tests...
2023-01-11T22:18:07.1722411Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1722702Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1722957Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1723171Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37346
2023-01-11T22:18:07.1723384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37347
2023-01-11T22:18:07.1723746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1723914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1724278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1724459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1724809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1724963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1725325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1725596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1725838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1726075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1726468Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1726852Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1727072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1727278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1727508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:18:07.1727745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:18:07.1728126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.1728508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.1728601Z ok (5.716s)
2023-01-11T22:18:07.1728620Z 
2023-01-11T22:18:07.1728875Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1728983Z Ran 1 test in 5.716s
2023-01-11T22:18:07.1729002Z 
2023-01-11T22:18:07.1729087Z OK
2023-01-11T22:18:07.1729106Z 
2023-01-11T22:18:07.1729211Z Generating XML reports...
2023-01-11T22:18:07.1729657Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220119.xml
2023-01-11T22:18:07.1730018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1730180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1730593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1730774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1730793Z 
2023-01-11T22:18:07.1730892Z Running tests...
2023-01-11T22:18:07.1731147Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1731446Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1731687Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1731897Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37470
2023-01-11T22:18:07.1732108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37471
2023-01-11T22:18:07.1732514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1732678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1733047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1733230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1733582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1733736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1734097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1734282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1734618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1734857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1735245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1735632Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1735849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1736067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1736197Z skip: Need at least 3 CUDA devices (4.280s)
2023-01-11T22:18:07.1736218Z 
2023-01-11T22:18:07.1736481Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1736592Z Ran 1 test in 4.280s
2023-01-11T22:18:07.1736611Z 
2023-01-11T22:18:07.1736714Z OK (skipped=1)
2023-01-11T22:18:07.1736733Z 
2023-01-11T22:18:07.1736850Z Generating XML reports...
2023-01-11T22:18:07.1737287Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220128.xml
2023-01-11T22:18:07.1737642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1737815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1738191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1738361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1738381Z 
2023-01-11T22:18:07.1738480Z Running tests...
2023-01-11T22:18:07.1738740Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1739053Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1739300Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2023-01-11T22:18:07.1739320Z 
2023-01-11T22:18:07.1739576Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1739676Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1739695Z 
2023-01-11T22:18:07.1739796Z OK (skipped=1)
2023-01-11T22:18:07.1739815Z 
2023-01-11T22:18:07.1739918Z Generating XML reports...
2023-01-11T22:18:07.1740356Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220134.xml
2023-01-11T22:18:07.1740717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1740885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1741266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1741452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1741471Z 
2023-01-11T22:18:07.1741574Z Running tests...
2023-01-11T22:18:07.1741832Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1742139Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1742371Z   test_barrier (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2023-01-11T22:18:07.1742390Z 
2023-01-11T22:18:07.1742647Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1742748Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1742767Z 
2023-01-11T22:18:07.1742866Z OK (skipped=1)
2023-01-11T22:18:07.1742885Z 
2023-01-11T22:18:07.1743003Z Generating XML reports...
2023-01-11T22:18:07.1743536Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220137.xml
2023-01-11T22:18:07.1743908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1744075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1744443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1744612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1744631Z 
2023-01-11T22:18:07.1744732Z Running tests...
2023-01-11T22:18:07.1744986Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1745289Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1745535Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1745752Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37641
2023-01-11T22:18:07.1745958Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37642
2023-01-11T22:18:07.1746321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1746475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1746846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1747025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1747382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1747546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1747910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1748096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1748334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1748574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1748953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1749340Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1749567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1749784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1749884Z ok (6.396s)
2023-01-11T22:18:07.1749904Z 
2023-01-11T22:18:07.1750160Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1750268Z Ran 1 test in 6.397s
2023-01-11T22:18:07.1750288Z 
2023-01-11T22:18:07.1750375Z OK
2023-01-11T22:18:07.1750394Z 
2023-01-11T22:18:07.1750499Z Generating XML reports...
2023-01-11T22:18:07.1750934Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220139.xml
2023-01-11T22:18:07.1751293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1751464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1752064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1752276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1752377Z 
2023-01-11T22:18:07.1752487Z Running tests...
2023-01-11T22:18:07.1752805Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1753124Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1753369Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2023-01-11T22:18:07.1753389Z 
2023-01-11T22:18:07.1753642Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1753747Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1753767Z 
2023-01-11T22:18:07.1753867Z OK (skipped=1)
2023-01-11T22:18:07.1753886Z 
2023-01-11T22:18:07.1754004Z Generating XML reports...
2023-01-11T22:18:07.1754442Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220148.xml
2023-01-11T22:18:07.1754805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1754984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1755348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1755519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1755538Z 
2023-01-11T22:18:07.1755638Z Running tests...
2023-01-11T22:18:07.1755891Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1756197Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1756452Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1756664Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37789
2023-01-11T22:18:07.1756871Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37790
2023-01-11T22:18:07.1757242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1757395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1757761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1757942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1758297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1758460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1758819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1759002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1759245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1759484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1759863Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1760254Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1760475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1760692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1760843Z skip: Skipped due to small world size. (4.166s)
2023-01-11T22:18:07.1760863Z 
2023-01-11T22:18:07.1761119Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1761226Z Ran 1 test in 4.167s
2023-01-11T22:18:07.1761299Z 
2023-01-11T22:18:07.1761410Z OK (skipped=1)
2023-01-11T22:18:07.1761466Z 
2023-01-11T22:18:07.1761585Z Generating XML reports...
2023-01-11T22:18:07.1762019Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220150.xml
2023-01-11T22:18:07.1762383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1762549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1762917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1763098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1763118Z 
2023-01-11T22:18:07.1763217Z Running tests...
2023-01-11T22:18:07.1763475Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1763779Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1764025Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support CPU barrier (0.002s)
2023-01-11T22:18:07.1764064Z 
2023-01-11T22:18:07.1764306Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1764408Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1764428Z 
2023-01-11T22:18:07.1764528Z OK (skipped=1)
2023-01-11T22:18:07.1764547Z 
2023-01-11T22:18:07.1764663Z Generating XML reports...
2023-01-11T22:18:07.1765101Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220157.xml
2023-01-11T22:18:07.1765462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1765626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1765994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1766173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1766208Z 
2023-01-11T22:18:07.1766298Z Running tests...
2023-01-11T22:18:07.1766555Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1766858Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1767109Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1767322Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37927
2023-01-11T22:18:07.1767530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37928
2023-01-11T22:18:07.1767891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1768044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1768421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1768606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1768961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1769130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1769498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1769677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1769913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1770148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1770626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1771024Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1771245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1771466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1771618Z skip: Skipped due to small world size. (4.267s)
2023-01-11T22:18:07.1771638Z 
2023-01-11T22:18:07.1771898Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1772006Z Ran 1 test in 4.268s
2023-01-11T22:18:07.1772026Z 
2023-01-11T22:18:07.1772129Z OK (skipped=1)
2023-01-11T22:18:07.1772148Z 
2023-01-11T22:18:07.1772267Z Generating XML reports...
2023-01-11T22:18:07.1772699Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220200.xml
2023-01-11T22:18:07.1773063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1773230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1773602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1773783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1773803Z 
2023-01-11T22:18:07.1773902Z Running tests...
2023-01-11T22:18:07.1774158Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1774465Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1774724Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2023-01-11T22:18:07.1774760Z 
2023-01-11T22:18:07.1775004Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1775108Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1775127Z 
2023-01-11T22:18:07.1775230Z OK (skipped=1)
2023-01-11T22:18:07.1775248Z 
2023-01-11T22:18:07.1775367Z Generating XML reports...
2023-01-11T22:18:07.1775808Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220206.xml
2023-01-11T22:18:07.1776168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1776336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1776703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1776874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1776905Z 
2023-01-11T22:18:07.1776994Z Running tests...
2023-01-11T22:18:07.1777252Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1777551Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1777817Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2023-01-11T22:18:07.1777836Z 
2023-01-11T22:18:07.1778088Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1778194Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1778213Z 
2023-01-11T22:18:07.1778314Z OK (skipped=1)
2023-01-11T22:18:07.1778333Z 
2023-01-11T22:18:07.1778450Z Generating XML reports...
2023-01-11T22:18:07.1778874Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220209.xml
2023-01-11T22:18:07.1779233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1779520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1779898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1780081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1780100Z 
2023-01-11T22:18:07.1780198Z Running tests...
2023-01-11T22:18:07.1780455Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1780756Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1781019Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... skip: Only gloo backend supports timeouts (0.002s)
2023-01-11T22:18:07.1781039Z 
2023-01-11T22:18:07.1781278Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1781382Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1781405Z 
2023-01-11T22:18:07.1781509Z OK (skipped=1)
2023-01-11T22:18:07.1781530Z 
2023-01-11T22:18:07.1781647Z Generating XML reports...
2023-01-11T22:18:07.1782082Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220211.xml
2023-01-11T22:18:07.1782438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1782607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1782977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1783147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1783175Z 
2023-01-11T22:18:07.1783266Z Running tests...
2023-01-11T22:18:07.1783520Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1783817Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1784074Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2023-01-11T22:18:07.1784094Z 
2023-01-11T22:18:07.1784345Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1784448Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1784467Z 
2023-01-11T22:18:07.1784566Z OK (skipped=1)
2023-01-11T22:18:07.1784585Z 
2023-01-11T22:18:07.1784698Z Generating XML reports...
2023-01-11T22:18:07.1785121Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220213.xml
2023-01-11T22:18:07.1785482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1785646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1786016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1786211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1786230Z 
2023-01-11T22:18:07.1786327Z Running tests...
2023-01-11T22:18:07.1786580Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1786883Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1787134Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... skip: GLOO Batch Send Recv CPU (0.002s)
2023-01-11T22:18:07.1787154Z 
2023-01-11T22:18:07.1787397Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1787502Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1787521Z 
2023-01-11T22:18:07.1787618Z OK (skipped=1)
2023-01-11T22:18:07.1787637Z 
2023-01-11T22:18:07.1787748Z Generating XML reports...
2023-01-11T22:18:07.1788178Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220216.xml
2023-01-11T22:18:07.1788635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1788809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1789183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1789367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1789387Z 
2023-01-11T22:18:07.1789475Z Running tests...
2023-01-11T22:18:07.1789732Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1790033Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1790308Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1790525Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38197
2023-01-11T22:18:07.1790735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38198
2023-01-11T22:18:07.1791094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1791260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1791620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1791802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1792407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1792574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1792937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1793126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1793362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1793598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1793990Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1794365Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1794583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1794803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1795032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:18:07.1795278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:18:07.1795665Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.1796048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.1796279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:18:07.1796510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:18:07.1796884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:18:07.1797267Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:18:07.1797448Z ok (4.379s)
2023-01-11T22:18:07.1797515Z 
2023-01-11T22:18:07.1797779Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1797884Z Ran 1 test in 4.379s
2023-01-11T22:18:07.1797904Z 
2023-01-11T22:18:07.1797988Z OK
2023-01-11T22:18:07.1798007Z 
2023-01-11T22:18:07.1798125Z Generating XML reports...
2023-01-11T22:18:07.1798572Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220218.xml
2023-01-11T22:18:07.1798932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1799087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1799460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1799643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1799665Z 
2023-01-11T22:18:07.1799769Z Running tests...
2023-01-11T22:18:07.1800024Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1800329Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1800585Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1800797Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38310
2023-01-11T22:18:07.1800995Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38311
2023-01-11T22:18:07.1801360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1801528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1801890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1802081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1802434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1802597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1802951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1803135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1803358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1803596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1803988Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1804379Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1804599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1804820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1804915Z ok (5.250s)
2023-01-11T22:18:07.1804935Z 
2023-01-11T22:18:07.1805195Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1805287Z Ran 1 test in 5.251s
2023-01-11T22:18:07.1805320Z 
2023-01-11T22:18:07.1805394Z OK
2023-01-11T22:18:07.1805412Z 
2023-01-11T22:18:07.1805528Z Generating XML reports...
2023-01-11T22:18:07.1805967Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220225.xml
2023-01-11T22:18:07.1806328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1806590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1806968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1807149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1807169Z 
2023-01-11T22:18:07.1807271Z Running tests...
2023-01-11T22:18:07.1807514Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1807815Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1808087Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1808292Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38426
2023-01-11T22:18:07.1808502Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38427
2023-01-11T22:18:07.1808868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1809036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1809407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1809576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1809935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1810103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1810460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1810642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1810890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1811133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1811526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1811913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1812122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1812344Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1812493Z skip: Skipped due to small world size. (4.316s)
2023-01-11T22:18:07.1812513Z 
2023-01-11T22:18:07.1812774Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1812882Z Ran 1 test in 4.317s
2023-01-11T22:18:07.1812902Z 
2023-01-11T22:18:07.1813010Z OK (skipped=1)
2023-01-11T22:18:07.1813030Z 
2023-01-11T22:18:07.1813146Z Generating XML reports...
2023-01-11T22:18:07.1813584Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220233.xml
2023-01-11T22:18:07.1813947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1814102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1814472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1814654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1814673Z 
2023-01-11T22:18:07.1814771Z Running tests...
2023-01-11T22:18:07.1815026Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1815427Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1815695Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1815903Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38531
2023-01-11T22:18:07.1816099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38532
2023-01-11T22:18:07.1816458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1816626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1816994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1817176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1817531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1817700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1818067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1818243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1818468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1818702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1819090Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1819475Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1819697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1819919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1820015Z ok (5.094s)
2023-01-11T22:18:07.1820034Z 
2023-01-11T22:18:07.1820293Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1820386Z Ran 1 test in 5.094s
2023-01-11T22:18:07.1820419Z 
2023-01-11T22:18:07.1820491Z OK
2023-01-11T22:18:07.1820510Z 
2023-01-11T22:18:07.1820627Z Generating XML reports...
2023-01-11T22:18:07.1821066Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220240.xml
2023-01-11T22:18:07.1821424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1821589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1821961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1822146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1822165Z 
2023-01-11T22:18:07.1822266Z Running tests...
2023-01-11T22:18:07.1822509Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1822814Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1823076Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1823285Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38637
2023-01-11T22:18:07.1823495Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38638
2023-01-11T22:18:07.1823856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1824023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1824491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1824683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1825030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1825194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1825553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1825736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1825975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1826207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1826604Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1826990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1827199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1827417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1827511Z ok (4.406s)
2023-01-11T22:18:07.1827531Z 
2023-01-11T22:18:07.1827786Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1827891Z Ran 1 test in 4.407s
2023-01-11T22:18:07.1827910Z 
2023-01-11T22:18:07.1827996Z OK
2023-01-11T22:18:07.1828015Z 
2023-01-11T22:18:07.1828128Z Generating XML reports...
2023-01-11T22:18:07.1828563Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220247.xml
2023-01-11T22:18:07.1828932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1829088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1829455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1829637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1829656Z 
2023-01-11T22:18:07.1829759Z Running tests...
2023-01-11T22:18:07.1830013Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1830314Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1830631Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1830850Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38742
2023-01-11T22:18:07.1831048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38743
2023-01-11T22:18:07.1831413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1831579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1832176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1832368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1832730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1832895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1833259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1833572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1833803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1834045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1834432Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1834818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1835040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1835260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1835356Z ok (5.180s)
2023-01-11T22:18:07.1835377Z 
2023-01-11T22:18:07.1835641Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1835736Z Ran 1 test in 5.180s
2023-01-11T22:18:07.1835772Z 
2023-01-11T22:18:07.1835845Z OK
2023-01-11T22:18:07.1835863Z 
2023-01-11T22:18:07.1835980Z Generating XML reports...
2023-01-11T22:18:07.1836426Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220254.xml
2023-01-11T22:18:07.1836786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1836951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1837318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1837499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1837518Z 
2023-01-11T22:18:07.1837617Z Running tests...
2023-01-11T22:18:07.1837866Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1838173Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1838434Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1838643Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38858
2023-01-11T22:18:07.1838854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38859
2023-01-11T22:18:07.1839213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1839384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1839757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1839943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1840291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1840460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1840821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1840993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1841228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1841462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1841858Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1842244Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1842559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1842785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1842882Z ok (5.874s)
2023-01-11T22:18:07.1842902Z 
2023-01-11T22:18:07.1843159Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1843263Z Ran 1 test in 5.875s
2023-01-11T22:18:07.1843282Z 
2023-01-11T22:18:07.1843367Z OK
2023-01-11T22:18:07.1843386Z 
2023-01-11T22:18:07.1843505Z Generating XML reports...
2023-01-11T22:18:07.1843951Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220302.xml
2023-01-11T22:18:07.1844311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1844466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1844843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1845024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1845044Z 
2023-01-11T22:18:07.1845145Z Running tests...
2023-01-11T22:18:07.1845401Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1845705Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1845970Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1846180Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38973
2023-01-11T22:18:07.1846375Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38974
2023-01-11T22:18:07.1846738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1846913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1847280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1847461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1847813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1847978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1848335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1848519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1848741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1848978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1849372Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1849760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1849983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1850202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1850296Z ok (4.379s)
2023-01-11T22:18:07.1850315Z 
2023-01-11T22:18:07.1850573Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1850665Z Ran 1 test in 4.379s
2023-01-11T22:18:07.1850697Z 
2023-01-11T22:18:07.1850770Z OK
2023-01-11T22:18:07.1850788Z 
2023-01-11T22:18:07.1850966Z Generating XML reports...
2023-01-11T22:18:07.1851451Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220310.xml
2023-01-11T22:18:07.1851821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1851992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1852363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1852543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1852562Z 
2023-01-11T22:18:07.1852662Z Running tests...
2023-01-11T22:18:07.1852903Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1853208Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1853462Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.1853484Z 
2023-01-11T22:18:07.1853737Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1853841Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1853860Z 
2023-01-11T22:18:07.1853960Z OK (skipped=1)
2023-01-11T22:18:07.1853979Z 
2023-01-11T22:18:07.1854090Z Generating XML reports...
2023-01-11T22:18:07.1854526Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220317.xml
2023-01-11T22:18:07.1854889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1855043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1855408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1855590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1855612Z 
2023-01-11T22:18:07.1855715Z Running tests...
2023-01-11T22:18:07.1855969Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1856277Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1856527Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1857268Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.675s)
2023-01-11T22:18:07.1857289Z 
2023-01-11T22:18:07.1857541Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1857633Z Ran 1 test in 1.675s
2023-01-11T22:18:07.1857667Z 
2023-01-11T22:18:07.1857760Z OK (skipped=1)
2023-01-11T22:18:07.1857779Z 
2023-01-11T22:18:07.1857896Z Generating XML reports...
2023-01-11T22:18:07.1858339Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220320.xml
2023-01-11T22:18:07.1858699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1858869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1859235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1859419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1859439Z 
2023-01-11T22:18:07.1859539Z Running tests...
2023-01-11T22:18:07.1859781Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1860087Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1860442Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.1860463Z 
2023-01-11T22:18:07.1860716Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1860823Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1860842Z 
2023-01-11T22:18:07.1860943Z OK (skipped=1)
2023-01-11T22:18:07.1860962Z 
2023-01-11T22:18:07.1861077Z Generating XML reports...
2023-01-11T22:18:07.1861511Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220324.xml
2023-01-11T22:18:07.1861866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1862022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1862390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1862580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1862600Z 
2023-01-11T22:18:07.1862698Z Running tests...
2023-01-11T22:18:07.1862953Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1863255Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1863508Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.1863528Z 
2023-01-11T22:18:07.1863778Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1863870Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1863901Z 
2023-01-11T22:18:07.1863988Z OK (skipped=1)
2023-01-11T22:18:07.1864006Z 
2023-01-11T22:18:07.1864123Z Generating XML reports...
2023-01-11T22:18:07.1864557Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220326.xml
2023-01-11T22:18:07.1864925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1865094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1865467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1865645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1865665Z 
2023-01-11T22:18:07.1865767Z Running tests...
2023-01-11T22:18:07.1866010Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1866311Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1866567Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... skip: NCCL broadcast multigpu skipped (0.002s)
2023-01-11T22:18:07.1866587Z 
2023-01-11T22:18:07.1866843Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1866947Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1866967Z 
2023-01-11T22:18:07.1867063Z OK (skipped=1)
2023-01-11T22:18:07.1867082Z 
2023-01-11T22:18:07.1867195Z Generating XML reports...
2023-01-11T22:18:07.1867632Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220329.xml
2023-01-11T22:18:07.1867994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1868149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1868516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1868695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1868715Z 
2023-01-11T22:18:07.1868812Z Running tests...
2023-01-11T22:18:07.1869164Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1869478Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1869734Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1870471Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.653s)
2023-01-11T22:18:07.1870491Z 
2023-01-11T22:18:07.1870745Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1870837Z Ran 1 test in 1.653s
2023-01-11T22:18:07.1870868Z 
2023-01-11T22:18:07.1870956Z OK (skipped=1)
2023-01-11T22:18:07.1870975Z 
2023-01-11T22:18:07.1871090Z Generating XML reports...
2023-01-11T22:18:07.1871530Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220331.xml
2023-01-11T22:18:07.1872108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1872295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1872672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1872855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1872875Z 
2023-01-11T22:18:07.1872977Z Running tests...
2023-01-11T22:18:07.1873221Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1873524Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1873833Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1874572Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85012 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.645s)
2023-01-11T22:18:07.1874592Z 
2023-01-11T22:18:07.1874844Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1874952Z Ran 1 test in 1.646s
2023-01-11T22:18:07.1874971Z 
2023-01-11T22:18:07.1875068Z OK (skipped=1)
2023-01-11T22:18:07.1875086Z 
2023-01-11T22:18:07.1875199Z Generating XML reports...
2023-01-11T22:18:07.1875638Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220335.xml
2023-01-11T22:18:07.1875997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1876160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1876532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1876713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1876733Z 
2023-01-11T22:18:07.1876836Z Running tests...
2023-01-11T22:18:07.1877089Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1877392Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1877700Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1878511Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85339 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.684s)
2023-01-11T22:18:07.1878580Z 
2023-01-11T22:18:07.1878847Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1878954Z Ran 1 test in 1.684s
2023-01-11T22:18:07.1878974Z 
2023-01-11T22:18:07.1879061Z OK (skipped=1)
2023-01-11T22:18:07.1879079Z 
2023-01-11T22:18:07.1879194Z Generating XML reports...
2023-01-11T22:18:07.1879633Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220339.xml
2023-01-11T22:18:07.1879988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1880153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1880523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1880717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1880737Z 
2023-01-11T22:18:07.1880837Z Running tests...
2023-01-11T22:18:07.1881079Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1881384Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1881652Z   test_ddp_apply_optim_in_backward (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1881863Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39346
2023-01-11T22:18:07.1882071Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39347
2023-01-11T22:18:07.1882432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1882593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1882974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1883156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1883503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1883670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1884031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1884211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1884449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1884685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1885075Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1885464Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1885688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1885897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1886675Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T22:18:07.1886779Z   warnings.warn(
2023-01-11T22:18:07.1887587Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T22:18:07.1887738Z   warnings.warn(
2023-01-11T22:18:07.1887970Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.1888197Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.1888421Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.1888643Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.1888857Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.1889078Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.1889298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.1889526Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.1889616Z ok (7.168s)
2023-01-11T22:18:07.1889636Z 
2023-01-11T22:18:07.1889897Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1890002Z Ran 1 test in 7.168s
2023-01-11T22:18:07.1890022Z 
2023-01-11T22:18:07.1890108Z OK
2023-01-11T22:18:07.1890127Z 
2023-01-11T22:18:07.1890241Z Generating XML reports...
2023-01-11T22:18:07.1890670Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220344.xml
2023-01-11T22:18:07.1891029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1891196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1891573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1891761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1891780Z 
2023-01-11T22:18:07.1891881Z Running tests...
2023-01-11T22:18:07.1892136Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1892437Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1892724Z   test_ddp_apply_optim_in_backward_grad_as_bucket_view_false (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1892933Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39464
2023-01-11T22:18:07.1893139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39465
2023-01-11T22:18:07.1893497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1893671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1894048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1894230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1894581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1894744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1895097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1895274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1895511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1895743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1896229Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1896623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1896848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1897067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1897831Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T22:18:07.1897938Z   warnings.warn(
2023-01-11T22:18:07.1898695Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T22:18:07.1898799Z   warnings.warn(
2023-01-11T22:18:07.1899027Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.1899254Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.1899482Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.1899709Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.1899803Z ok (6.336s)
2023-01-11T22:18:07.1899824Z 
2023-01-11T22:18:07.1900085Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1900182Z Ran 1 test in 6.337s
2023-01-11T22:18:07.1900211Z 
2023-01-11T22:18:07.1900288Z OK
2023-01-11T22:18:07.1900306Z 
2023-01-11T22:18:07.1900422Z Generating XML reports...
2023-01-11T22:18:07.1900867Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220353.xml
2023-01-11T22:18:07.1901232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1901399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1901768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1901954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1901973Z 
2023-01-11T22:18:07.1902074Z Running tests...
2023-01-11T22:18:07.1902318Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1902629Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1902916Z   test_ddp_apply_optim_in_backward_ignored_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1903124Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39582
2023-01-11T22:18:07.1903335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39583
2023-01-11T22:18:07.1903697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1903861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1904228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1904396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1904804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1905025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1905387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1905563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1905801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1906033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1906422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1906804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1907020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1907239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1908009Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T22:18:07.1908113Z   warnings.warn(
2023-01-11T22:18:07.1908873Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T22:18:07.1908974Z   warnings.warn(
2023-01-11T22:18:07.1909073Z ok (6.391s)
2023-01-11T22:18:07.1909093Z 
2023-01-11T22:18:07.1909350Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1909452Z Ran 1 test in 6.391s
2023-01-11T22:18:07.1909471Z 
2023-01-11T22:18:07.1909552Z OK
2023-01-11T22:18:07.1909571Z 
2023-01-11T22:18:07.1909676Z Generating XML reports...
2023-01-11T22:18:07.1910119Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220402.xml
2023-01-11T22:18:07.1910481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1910646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1911017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1911200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1911223Z 
2023-01-11T22:18:07.1911324Z Running tests...
2023-01-11T22:18:07.1911582Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1912109Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1912370Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1912581Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39702
2023-01-11T22:18:07.1912792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39703
2023-01-11T22:18:07.1913157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1913325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1913694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1914008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1914372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1914528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1914894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1915078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1915317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1915553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1915942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1916338Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1916560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1916779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1916994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.1917213Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.1917308Z ok (5.675s)
2023-01-11T22:18:07.1917330Z 
2023-01-11T22:18:07.1917592Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1917697Z Ran 1 test in 5.676s
2023-01-11T22:18:07.1917716Z 
2023-01-11T22:18:07.1917798Z OK
2023-01-11T22:18:07.1917818Z 
2023-01-11T22:18:07.1917933Z Generating XML reports...
2023-01-11T22:18:07.1918375Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220411.xml
2023-01-11T22:18:07.1918725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1918898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1919270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1919453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1919473Z 
2023-01-11T22:18:07.1919573Z Running tests...
2023-01-11T22:18:07.1919830Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1920133Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1920401Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1920613Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39820
2023-01-11T22:18:07.1920812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39821
2023-01-11T22:18:07.1921174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1921341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1921709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1921890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1922244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1922405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1922817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1923072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1923311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1923544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1923943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1924327Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1924549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1924763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1924991Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.1925222Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.1925434Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.1925654Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.1925747Z ok (5.682s)
2023-01-11T22:18:07.1925766Z 
2023-01-11T22:18:07.1926029Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1926134Z Ran 1 test in 5.682s
2023-01-11T22:18:07.1926153Z 
2023-01-11T22:18:07.1926236Z OK
2023-01-11T22:18:07.1926255Z 
2023-01-11T22:18:07.1926370Z Generating XML reports...
2023-01-11T22:18:07.1926811Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220419.xml
2023-01-11T22:18:07.1927159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1927338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1927704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1927883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1927902Z 
2023-01-11T22:18:07.1928004Z Running tests...
2023-01-11T22:18:07.1928262Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1928563Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1928823Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1929565Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.633s)
2023-01-11T22:18:07.1929588Z 
2023-01-11T22:18:07.1929848Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1929941Z Ran 1 test in 1.633s
2023-01-11T22:18:07.1929960Z 
2023-01-11T22:18:07.1930059Z OK (skipped=1)
2023-01-11T22:18:07.1930078Z 
2023-01-11T22:18:07.1931383Z Generating XML reports...
2023-01-11T22:18:07.1931836Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220428.xml
2023-01-11T22:18:07.1932195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1932360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1932730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1933012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1933033Z 
2023-01-11T22:18:07.1933136Z Running tests...
2023-01-11T22:18:07.1933381Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1933684Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1933969Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1934706Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.667s)
2023-01-11T22:18:07.1934726Z 
2023-01-11T22:18:07.1934980Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1935085Z Ran 1 test in 1.667s
2023-01-11T22:18:07.1935108Z 
2023-01-11T22:18:07.1935204Z OK (skipped=1)
2023-01-11T22:18:07.1935222Z 
2023-01-11T22:18:07.1935340Z Generating XML reports...
2023-01-11T22:18:07.1935788Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220432.xml
2023-01-11T22:18:07.1936145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1936301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1936676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1936857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1936877Z 
2023-01-11T22:18:07.1936976Z Running tests...
2023-01-11T22:18:07.1937232Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1937549Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1937831Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1938043Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40006
2023-01-11T22:18:07.1938244Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40007
2023-01-11T22:18:07.1938607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1938772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1939145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1939322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1939682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1939848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1940212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1940389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1940617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1940858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1941254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1941645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1941968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1942194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1942394Z <class 'torch.nn.parameter.Parameter'>
2023-01-11T22:18:07.1942488Z ok (5.640s)
2023-01-11T22:18:07.1942508Z 
2023-01-11T22:18:07.1942752Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1942855Z Ran 1 test in 5.641s
2023-01-11T22:18:07.1942875Z 
2023-01-11T22:18:07.1942963Z OK
2023-01-11T22:18:07.1942982Z 
2023-01-11T22:18:07.1943098Z Generating XML reports...
2023-01-11T22:18:07.1943545Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220436.xml
2023-01-11T22:18:07.1943911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1944082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1944459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1944647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1944666Z 
2023-01-11T22:18:07.1944756Z Running tests...
2023-01-11T22:18:07.1945012Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1945316Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1945621Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1945838Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40121
2023-01-11T22:18:07.1946051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40122
2023-01-11T22:18:07.1946415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1946588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1946947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1947130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1947485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1947653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1948016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1948192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1948433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1948675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1949068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1949448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1949668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1949890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1949983Z ok (5.581s)
2023-01-11T22:18:07.1950003Z 
2023-01-11T22:18:07.1950256Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1950358Z Ran 1 test in 5.582s
2023-01-11T22:18:07.1950378Z 
2023-01-11T22:18:07.1950459Z OK
2023-01-11T22:18:07.1950478Z 
2023-01-11T22:18:07.1950649Z Generating XML reports...
2023-01-11T22:18:07.1951139Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220444.xml
2023-01-11T22:18:07.1951496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1951665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1952381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1952575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1952594Z 
2023-01-11T22:18:07.1952694Z Running tests...
2023-01-11T22:18:07.1952959Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1953267Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1953529Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1953735Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40236
2023-01-11T22:18:07.1953951Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40237
2023-01-11T22:18:07.1954315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1954484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1954850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1955038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1955393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1955554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1955926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1956096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1956337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1956576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1956969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1957357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1957577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1957795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1958033Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.1958268Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.1958351Z ok (6.053s)
2023-01-11T22:18:07.1958371Z 
2023-01-11T22:18:07.1958632Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1958733Z Ran 1 test in 6.053s
2023-01-11T22:18:07.1958752Z 
2023-01-11T22:18:07.1958835Z OK
2023-01-11T22:18:07.1958854Z 
2023-01-11T22:18:07.1958975Z Generating XML reports...
2023-01-11T22:18:07.1959422Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220452.xml
2023-01-11T22:18:07.1959782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1959947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1960463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1960653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1960673Z 
2023-01-11T22:18:07.1960777Z Running tests...
2023-01-11T22:18:07.1961032Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1961341Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1961628Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1961841Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40355
2023-01-11T22:18:07.1962047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40356
2023-01-11T22:18:07.1962397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1962575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1962942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1963123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1963474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1963640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1964009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1964192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1964427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1964661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1965060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1965450Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1965674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1965891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1966664Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:18:07.1966766Z ok (5.682s)
2023-01-11T22:18:07.1966787Z 
2023-01-11T22:18:07.1967039Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1967142Z Ran 1 test in 5.682s
2023-01-11T22:18:07.1967162Z 
2023-01-11T22:18:07.1967246Z OK
2023-01-11T22:18:07.1967264Z 
2023-01-11T22:18:07.1967370Z Generating XML reports...
2023-01-11T22:18:07.1967815Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220501.xml
2023-01-11T22:18:07.1968178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1968342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1968764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1969007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1969027Z 
2023-01-11T22:18:07.1969136Z Running tests...
2023-01-11T22:18:07.1969388Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1969693Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1969960Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1970701Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78235 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.704s)
2023-01-11T22:18:07.1970729Z 
2023-01-11T22:18:07.1970981Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1971088Z Ran 1 test in 1.704s
2023-01-11T22:18:07.1971108Z 
2023-01-11T22:18:07.1971209Z OK (skipped=1)
2023-01-11T22:18:07.1971229Z 
2023-01-11T22:18:07.1971343Z Generating XML reports...
2023-01-11T22:18:07.1971788Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220509.xml
2023-01-11T22:18:07.1972149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1972315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1972683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1972853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1972872Z 
2023-01-11T22:18:07.1972971Z Running tests...
2023-01-11T22:18:07.1973229Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1973531Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1973853Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... skip: Gloo-only test (0.002s)
2023-01-11T22:18:07.1973872Z 
2023-01-11T22:18:07.1974120Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1974223Z Ran 1 test in 0.002s
2023-01-11T22:18:07.1974243Z 
2023-01-11T22:18:07.1974337Z OK (skipped=1)
2023-01-11T22:18:07.1974356Z 
2023-01-11T22:18:07.1974460Z Generating XML reports...
2023-01-11T22:18:07.1974900Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220513.xml
2023-01-11T22:18:07.1975259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1975422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1975800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1975983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1976002Z 
2023-01-11T22:18:07.1976100Z Running tests...
2023-01-11T22:18:07.1976351Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1976657Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1976891Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1977630Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77324 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.696s)
2023-01-11T22:18:07.1977700Z 
2023-01-11T22:18:07.1977998Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1978104Z Ran 1 test in 1.696s
2023-01-11T22:18:07.1978123Z 
2023-01-11T22:18:07.1978221Z OK (skipped=1)
2023-01-11T22:18:07.1978240Z 
2023-01-11T22:18:07.1978354Z Generating XML reports...
2023-01-11T22:18:07.1978796Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220516.xml
2023-01-11T22:18:07.1979159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1979325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1979698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1979870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1979892Z 
2023-01-11T22:18:07.1979992Z Running tests...
2023-01-11T22:18:07.1980252Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1980558Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1980825Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1981037Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40574
2023-01-11T22:18:07.1981248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40575
2023-01-11T22:18:07.1981611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1981765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1982128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1982320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1982676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1982836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1983195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1983379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1983616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.1983854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.1984237Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1984635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.1984861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.1985087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.1985865Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1331: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2023-01-11T22:18:07.1986186Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2023-01-11T22:18:07.1987007Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1331: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2023-01-11T22:18:07.1987373Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2023-01-11T22:18:07.1987468Z ok (6.189s)
2023-01-11T22:18:07.1987488Z 
2023-01-11T22:18:07.1987744Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1987837Z Ran 1 test in 6.189s
2023-01-11T22:18:07.1987869Z 
2023-01-11T22:18:07.1987942Z OK
2023-01-11T22:18:07.1987960Z 
2023-01-11T22:18:07.1988077Z Generating XML reports...
2023-01-11T22:18:07.1988518Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220520.xml
2023-01-11T22:18:07.1988880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1989055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1989422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1989605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1989625Z 
2023-01-11T22:18:07.1989726Z Running tests...
2023-01-11T22:18:07.1989968Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1990277Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1990543Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1991275Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.665s)
2023-01-11T22:18:07.1991301Z 
2023-01-11T22:18:07.1991562Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1991662Z Ran 1 test in 1.665s
2023-01-11T22:18:07.1991681Z 
2023-01-11T22:18:07.1991780Z OK (skipped=1)
2023-01-11T22:18:07.1991799Z 
2023-01-11T22:18:07.1992140Z Generating XML reports...
2023-01-11T22:18:07.1992601Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220529.xml
2023-01-11T22:18:07.1992964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1993120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1993492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1993671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1993696Z 
2023-01-11T22:18:07.1993801Z Running tests...
2023-01-11T22:18:07.1994052Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1994357Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1994622Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1995361Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.643s)
2023-01-11T22:18:07.1995381Z 
2023-01-11T22:18:07.1995634Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1995726Z Ran 1 test in 1.643s
2023-01-11T22:18:07.1995758Z 
2023-01-11T22:18:07.1995934Z OK (skipped=1)
2023-01-11T22:18:07.1995953Z 
2023-01-11T22:18:07.1996123Z Generating XML reports...
2023-01-11T22:18:07.1996579Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220533.xml
2023-01-11T22:18:07.1996943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1997110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1997480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.1997658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.1997678Z 
2023-01-11T22:18:07.1997782Z Running tests...
2023-01-11T22:18:07.1998023Z ----------------------------------------------------------------------
2023-01-11T22:18:07.1998328Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.1998624Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.1998844Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40791
2023-01-11T22:18:07.1999049Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40792
2023-01-11T22:18:07.1999406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.1999575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.1999942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2000113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2000463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2000634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2000998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2001176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2001417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2001654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2002046Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2002430Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2002645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2002886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:18:07.2003100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2003335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:18:07.2003725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.2004113Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.2004346Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2004573Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2004799Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2005115Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2005217Z ok (6.447s)
2023-01-11T22:18:07.2005237Z 
2023-01-11T22:18:07.2005494Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2005599Z Ran 1 test in 6.447s
2023-01-11T22:18:07.2005618Z 
2023-01-11T22:18:07.2005703Z OK
2023-01-11T22:18:07.2005722Z 
2023-01-11T22:18:07.2005833Z Generating XML reports...
2023-01-11T22:18:07.2006270Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220537.xml
2023-01-11T22:18:07.2006631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2006798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2007158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2007343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2007362Z 
2023-01-11T22:18:07.2007464Z Running tests...
2023-01-11T22:18:07.2007720Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2008023Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2008294Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2008510Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40912
2023-01-11T22:18:07.2008718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40913
2023-01-11T22:18:07.2009072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2009237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2009618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2009800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2010152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2010319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2010679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2010862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2011089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2011328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2011728Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2012119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2012339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2012609Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2023-01-11T22:18:07.2012829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2013089Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2023-01-11T22:18:07.2013318Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2013537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2013848Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2014082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2014352Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2023-01-11T22:18:07.2014620Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2023-01-11T22:18:07.2014888Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2023-01-11T22:18:07.2015144Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2023-01-11T22:18:07.2015369Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2015595Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2015816Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2016039Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2016301Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2023-01-11T22:18:07.2016570Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2023-01-11T22:18:07.2016839Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2023-01-11T22:18:07.2017099Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2023-01-11T22:18:07.2017324Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2017548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2017782Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2017994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2018089Z ok (6.853s)
2023-01-11T22:18:07.2018109Z 
2023-01-11T22:18:07.2018372Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2018477Z Ran 1 test in 6.853s
2023-01-11T22:18:07.2018496Z 
2023-01-11T22:18:07.2018581Z OK
2023-01-11T22:18:07.2018600Z 
2023-01-11T22:18:07.2018714Z Generating XML reports...
2023-01-11T22:18:07.2019158Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220546.xml
2023-01-11T22:18:07.2019518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2019689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2020058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2020244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2020263Z 
2023-01-11T22:18:07.2020364Z Running tests...
2023-01-11T22:18:07.2020619Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2020920Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2021190Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2021926Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.683s)
2023-01-11T22:18:07.2021995Z 
2023-01-11T22:18:07.2022296Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2022407Z Ran 1 test in 1.683s
2023-01-11T22:18:07.2022427Z 
2023-01-11T22:18:07.2022515Z OK (skipped=1)
2023-01-11T22:18:07.2022547Z 
2023-01-11T22:18:07.2022652Z Generating XML reports...
2023-01-11T22:18:07.2023098Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220555.xml
2023-01-11T22:18:07.2023459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2023627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2024001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2024181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2024203Z 
2023-01-11T22:18:07.2024303Z Running tests...
2023-01-11T22:18:07.2024561Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2024860Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2025134Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2025348Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41065
2023-01-11T22:18:07.2025553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41066
2023-01-11T22:18:07.2025916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2026085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2026453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2026645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2026989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2027152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2027516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2027701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2027942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2028183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2028575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2028974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2029200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2029753Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:18:07.2029966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2030545Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:18:07.2030891Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2031125Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2031394Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2023-01-11T22:18:07.2031666Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2023-01-11T22:18:07.2032188Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2023-01-11T22:18:07.2032528Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2023-01-11T22:18:07.2032851Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2023-01-11T22:18:07.2033178Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2023-01-11T22:18:07.2033504Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2023-01-11T22:18:07.2033811Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2023-01-11T22:18:07.2034044Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2034275Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2034370Z ok (6.148s)
2023-01-11T22:18:07.2034391Z 
2023-01-11T22:18:07.2034666Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2034772Z Ran 1 test in 6.148s
2023-01-11T22:18:07.2034795Z 
2023-01-11T22:18:07.2034882Z OK
2023-01-11T22:18:07.2034901Z 
2023-01-11T22:18:07.2035017Z Generating XML reports...
2023-01-11T22:18:07.2035453Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220559.xml
2023-01-11T22:18:07.2035822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2035988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2036359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2036544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2036563Z 
2023-01-11T22:18:07.2036665Z Running tests...
2023-01-11T22:18:07.2036918Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2037232Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2037623Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T22:18:07.2037643Z 
2023-01-11T22:18:07.2037896Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2037989Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2038008Z 
2023-01-11T22:18:07.2038111Z OK (skipped=1)
2023-01-11T22:18:07.2038130Z 
2023-01-11T22:18:07.2038251Z Generating XML reports...
2023-01-11T22:18:07.2038691Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220608.xml
2023-01-11T22:18:07.2039054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2039219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2039740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2039930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2039949Z 
2023-01-11T22:18:07.2040039Z Running tests...
2023-01-11T22:18:07.2040291Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2040595Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2040981Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T22:18:07.2041000Z 
2023-01-11T22:18:07.2041253Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2041355Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2041378Z 
2023-01-11T22:18:07.2041477Z OK (skipped=1)
2023-01-11T22:18:07.2041498Z 
2023-01-11T22:18:07.2041613Z Generating XML reports...
2023-01-11T22:18:07.2042060Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220611.xml
2023-01-11T22:18:07.2042410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2042577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2042948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2043127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2043146Z 
2023-01-11T22:18:07.2043247Z Running tests...
2023-01-11T22:18:07.2043502Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2043809Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2044260Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T22:18:07.2044280Z 
2023-01-11T22:18:07.2044535Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2044639Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2044658Z 
2023-01-11T22:18:07.2044745Z OK (skipped=1)
2023-01-11T22:18:07.2044763Z 
2023-01-11T22:18:07.2044881Z Generating XML reports...
2023-01-11T22:18:07.2045323Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220613.xml
2023-01-11T22:18:07.2045683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2045857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2046237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2046418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2046437Z 
2023-01-11T22:18:07.2046541Z Running tests...
2023-01-11T22:18:07.2046784Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2047090Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2047525Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T22:18:07.2047546Z 
2023-01-11T22:18:07.2047797Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2047963Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2047982Z 
2023-01-11T22:18:07.2048123Z OK (skipped=1)
2023-01-11T22:18:07.2048143Z 
2023-01-11T22:18:07.2048262Z Generating XML reports...
2023-01-11T22:18:07.2048704Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220615.xml
2023-01-11T22:18:07.2049065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2049237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2049596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2049777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2049797Z 
2023-01-11T22:18:07.2049897Z Running tests...
2023-01-11T22:18:07.2050150Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2050463Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2050903Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T22:18:07.2050923Z 
2023-01-11T22:18:07.2051181Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2051287Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2051306Z 
2023-01-11T22:18:07.2051403Z OK (skipped=1)
2023-01-11T22:18:07.2051422Z 
2023-01-11T22:18:07.2051526Z Generating XML reports...
2023-01-11T22:18:07.2051968Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220618.xml
2023-01-11T22:18:07.2052329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2052506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2052876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2053057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2053076Z 
2023-01-11T22:18:07.2053177Z Running tests...
2023-01-11T22:18:07.2053433Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2053741Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2054167Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T22:18:07.2054193Z 
2023-01-11T22:18:07.2054431Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2054542Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2054562Z 
2023-01-11T22:18:07.2054662Z OK (skipped=1)
2023-01-11T22:18:07.2054681Z 
2023-01-11T22:18:07.2054796Z Generating XML reports...
2023-01-11T22:18:07.2055234Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220620.xml
2023-01-11T22:18:07.2055597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2055761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2056127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2056296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2056327Z 
2023-01-11T22:18:07.2056415Z Running tests...
2023-01-11T22:18:07.2056664Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2057064Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2057507Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T22:18:07.2057527Z 
2023-01-11T22:18:07.2057784Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2057889Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2057908Z 
2023-01-11T22:18:07.2058009Z OK (skipped=1)
2023-01-11T22:18:07.2058028Z 
2023-01-11T22:18:07.2058145Z Generating XML reports...
2023-01-11T22:18:07.2058572Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220623.xml
2023-01-11T22:18:07.2058934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2059109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2059483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2059664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2059684Z 
2023-01-11T22:18:07.2059782Z Running tests...
2023-01-11T22:18:07.2060037Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2060349Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2060782Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T22:18:07.2060802Z 
2023-01-11T22:18:07.2061060Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2061153Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2061173Z 
2023-01-11T22:18:07.2061274Z OK (skipped=1)
2023-01-11T22:18:07.2061293Z 
2023-01-11T22:18:07.2061407Z Generating XML reports...
2023-01-11T22:18:07.2061836Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220625.xml
2023-01-11T22:18:07.2062195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2062364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2062733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2062913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2062933Z 
2023-01-11T22:18:07.2063030Z Running tests...
2023-01-11T22:18:07.2063278Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2063588Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2064022Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T22:18:07.2064041Z 
2023-01-11T22:18:07.2064288Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2064390Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2064408Z 
2023-01-11T22:18:07.2064506Z OK (skipped=1)
2023-01-11T22:18:07.2064525Z 
2023-01-11T22:18:07.2064636Z Generating XML reports...
2023-01-11T22:18:07.2065074Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220627.xml
2023-01-11T22:18:07.2065519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2065720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2066093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2066278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2066298Z 
2023-01-11T22:18:07.2066394Z Running tests...
2023-01-11T22:18:07.2066646Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2066952Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2067384Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T22:18:07.2067408Z 
2023-01-11T22:18:07.2067668Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2067773Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2067792Z 
2023-01-11T22:18:07.2067879Z OK (skipped=1)
2023-01-11T22:18:07.2067898Z 
2023-01-11T22:18:07.2068013Z Generating XML reports...
2023-01-11T22:18:07.2068451Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220630.xml
2023-01-11T22:18:07.2068811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2068977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2069347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2069528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2069547Z 
2023-01-11T22:18:07.2069655Z Running tests...
2023-01-11T22:18:07.2069911Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2070208Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2070587Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T22:18:07.2070608Z 
2023-01-11T22:18:07.2070862Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2070965Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2070984Z 
2023-01-11T22:18:07.2071081Z OK (skipped=1)
2023-01-11T22:18:07.2071101Z 
2023-01-11T22:18:07.2071212Z Generating XML reports...
2023-01-11T22:18:07.2071648Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220632.xml
2023-01-11T22:18:07.2072254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2072436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2072805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2072990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2073009Z 
2023-01-11T22:18:07.2073111Z Running tests...
2023-01-11T22:18:07.2073363Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2073665Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2074045Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... skip: Issues with async error handling, see https://github.com/pytorch/pytorch/issues/73259 (0.002s)
2023-01-11T22:18:07.2074064Z 
2023-01-11T22:18:07.2074452Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2074564Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2074583Z 
2023-01-11T22:18:07.2074682Z OK (skipped=1)
2023-01-11T22:18:07.2074701Z 
2023-01-11T22:18:07.2074805Z Generating XML reports...
2023-01-11T22:18:07.2075250Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220634.xml
2023-01-11T22:18:07.2075612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2075782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2076156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2076336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2076356Z 
2023-01-11T22:18:07.2076455Z Running tests...
2023-01-11T22:18:07.2076711Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2077006Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2077270Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2078014Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.697s)
2023-01-11T22:18:07.2078034Z 
2023-01-11T22:18:07.2078288Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2078388Z Ran 1 test in 1.697s
2023-01-11T22:18:07.2078407Z 
2023-01-11T22:18:07.2078510Z OK (skipped=1)
2023-01-11T22:18:07.2078528Z 
2023-01-11T22:18:07.2078644Z Generating XML reports...
2023-01-11T22:18:07.2079088Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220637.xml
2023-01-11T22:18:07.2079449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2079615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2079975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2080159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2080179Z 
2023-01-11T22:18:07.2080277Z Running tests...
2023-01-11T22:18:07.2080529Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2080836Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2081088Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2081305Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41614
2023-01-11T22:18:07.2081516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41615
2023-01-11T22:18:07.2081881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2082037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2082403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2082587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2082941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2083107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2083515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2083740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2083981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2084207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2084606Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2084994Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2085219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2085437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2085535Z ok (5.876s)
2023-01-11T22:18:07.2085555Z 
2023-01-11T22:18:07.2085820Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2085920Z Ran 1 test in 5.877s
2023-01-11T22:18:07.2085939Z 
2023-01-11T22:18:07.2086027Z OK
2023-01-11T22:18:07.2086045Z 
2023-01-11T22:18:07.2086150Z Generating XML reports...
2023-01-11T22:18:07.2086592Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220641.xml
2023-01-11T22:18:07.2086953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2087120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2087490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2087674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2087697Z 
2023-01-11T22:18:07.2087800Z Running tests...
2023-01-11T22:18:07.2088056Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2088352Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2088627Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2088840Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41728
2023-01-11T22:18:07.2089047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41729
2023-01-11T22:18:07.2089409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2089575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2089946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2090134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2090492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2090645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2091018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2091196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2091436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2091669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2092067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2092571Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2092804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2093022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2093241Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2093469Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2093863Z /opt/conda/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmph2mvjm2n'>
2023-01-11T22:18:07.2094020Z   _warnings.warn(warn_message, ResourceWarning)
2023-01-11T22:18:07.2094406Z /opt/conda/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpqejkoujc'>
2023-01-11T22:18:07.2094561Z   _warnings.warn(warn_message, ResourceWarning)
2023-01-11T22:18:07.2094662Z ok (6.085s)
2023-01-11T22:18:07.2094683Z 
2023-01-11T22:18:07.2094938Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2095030Z Ran 1 test in 6.085s
2023-01-11T22:18:07.2095049Z 
2023-01-11T22:18:07.2095133Z OK
2023-01-11T22:18:07.2095152Z 
2023-01-11T22:18:07.2095268Z Generating XML reports...
2023-01-11T22:18:07.2095714Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220650.xml
2023-01-11T22:18:07.2096068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2096239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2096605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2096784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2096806Z 
2023-01-11T22:18:07.2096914Z Running tests...
2023-01-11T22:18:07.2097157Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2097464Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2097732Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.005s)
2023-01-11T22:18:07.2097751Z 
2023-01-11T22:18:07.2098005Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2098109Z Ran 1 test in 0.005s
2023-01-11T22:18:07.2098128Z 
2023-01-11T22:18:07.2098225Z OK (skipped=1)
2023-01-11T22:18:07.2098244Z 
2023-01-11T22:18:07.2098358Z Generating XML reports...
2023-01-11T22:18:07.2098802Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220658.xml
2023-01-11T22:18:07.2099153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2099327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2099699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2099881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2099900Z 
2023-01-11T22:18:07.2099998Z Running tests...
2023-01-11T22:18:07.2100248Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2100556Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2100812Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2101024Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41880
2023-01-11T22:18:07.2101224Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41881
2023-01-11T22:18:07.2101680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2101854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2102223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2102406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2102760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2102923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2103285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2103452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2103706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2103943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2104339Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2104729Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2104947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2105170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2105401Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2105629Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2105716Z ok (6.181s)
2023-01-11T22:18:07.2105738Z 
2023-01-11T22:18:07.2105994Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2106096Z Ran 1 test in 6.181s
2023-01-11T22:18:07.2106116Z 
2023-01-11T22:18:07.2106201Z OK
2023-01-11T22:18:07.2106221Z 
2023-01-11T22:18:07.2106332Z Generating XML reports...
2023-01-11T22:18:07.2106774Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220701.xml
2023-01-11T22:18:07.2107131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2107295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2107665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2107836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2107859Z 
2023-01-11T22:18:07.2107967Z Running tests...
2023-01-11T22:18:07.2108214Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2108517Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2108796Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2109009Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41999
2023-01-11T22:18:07.2109222Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42000
2023-01-11T22:18:07.2109584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2109742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2110114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2110394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2110755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2110921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2111285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2111463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2111702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2112160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2112559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2112961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2113186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2113407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2113637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:18:07.2113871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:18:07.2114261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.2114646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.2114883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:18:07.2115108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:18:07.2115491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:18:07.2115879Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:18:07.2115975Z ok (5.306s)
2023-01-11T22:18:07.2115995Z 
2023-01-11T22:18:07.2116256Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2116359Z Ran 1 test in 5.306s
2023-01-11T22:18:07.2116378Z 
2023-01-11T22:18:07.2116464Z OK
2023-01-11T22:18:07.2116483Z 
2023-01-11T22:18:07.2116604Z Generating XML reports...
2023-01-11T22:18:07.2117037Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220709.xml
2023-01-11T22:18:07.2117408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2117579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2117953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2118133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2118153Z 
2023-01-11T22:18:07.2118251Z Running tests...
2023-01-11T22:18:07.2118501Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2118811Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2119083Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2119284Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42121
2023-01-11T22:18:07.2119625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42122
2023-01-11T22:18:07.2120000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2120167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2120536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2120717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2121071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2121233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2121600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2121775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2122014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2122255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2122647Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2123032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2123259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2123477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2123710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:18:07.2123940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:18:07.2124335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.2124721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.2124955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:18:07.2125184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:18:07.2125562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:18:07.2125948Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:18:07.2126419Z [W ProcessGroupNCCL.cpp:950] [Rank 0] Found key in store: NCCLABORTEDCOMM:20e517ac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2023-01-11T22:18:07.2126882Z [W ProcessGroupNCCL.cpp:950] [Rank 1] Found key in store: NCCLABORTEDCOMM:20e517ac1102000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, from rank: 0. This means that rank has aborted its NCCL communicators previously and is not in a healthy state.. Aborting appropriate communicators
2023-01-11T22:18:07.2126977Z ok (15.385s)
2023-01-11T22:18:07.2126997Z 
2023-01-11T22:18:07.2127256Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2127358Z Ran 1 test in 15.386s
2023-01-11T22:18:07.2127378Z 
2023-01-11T22:18:07.2127501Z OK
2023-01-11T22:18:07.2127520Z 
2023-01-11T22:18:07.2127674Z Generating XML reports...
2023-01-11T22:18:07.2128125Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220717.xml
2023-01-11T22:18:07.2128484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2128651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2129022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2129207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2129227Z 
2023-01-11T22:18:07.2129331Z Running tests...
2023-01-11T22:18:07.2129576Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2129885Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2130198Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2130417Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42243
2023-01-11T22:18:07.2130670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42244
2023-01-11T22:18:07.2131038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2131205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2131575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2131755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2132099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2132278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2132641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2132819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2133062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2133297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2133689Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2134076Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2134297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2134510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2134604Z ok (6.282s)
2023-01-11T22:18:07.2134623Z 
2023-01-11T22:18:07.2134885Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2134991Z Ran 1 test in 6.283s
2023-01-11T22:18:07.2135011Z 
2023-01-11T22:18:07.2135098Z OK
2023-01-11T22:18:07.2135117Z 
2023-01-11T22:18:07.2135234Z Generating XML reports...
2023-01-11T22:18:07.2135673Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220735.xml
2023-01-11T22:18:07.2136035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2136192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2136563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2136840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2136861Z 
2023-01-11T22:18:07.2136969Z Running tests...
2023-01-11T22:18:07.2137225Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2137529Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2137816Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2138030Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42362
2023-01-11T22:18:07.2138242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42363
2023-01-11T22:18:07.2138595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2138765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2139141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2139324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2139676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2139841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2140211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2140391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2140617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2140857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2141254Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2141649Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2141874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2142098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2142190Z ok (6.247s)
2023-01-11T22:18:07.2142210Z 
2023-01-11T22:18:07.2142468Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2142573Z Ran 1 test in 6.247s
2023-01-11T22:18:07.2142592Z 
2023-01-11T22:18:07.2142664Z OK
2023-01-11T22:18:07.2142683Z 
2023-01-11T22:18:07.2142801Z Generating XML reports...
2023-01-11T22:18:07.2143244Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220744.xml
2023-01-11T22:18:07.2143612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2143783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2144156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2144338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2144357Z 
2023-01-11T22:18:07.2144462Z Running tests...
2023-01-11T22:18:07.2144706Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2145009Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2145259Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2145466Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42481
2023-01-11T22:18:07.2145770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42482
2023-01-11T22:18:07.2146139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2146308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2146676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2146859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2147203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2147365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2147731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2147918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2148156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2148395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2148789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2149177Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2149399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2149612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2149708Z ok (6.091s)
2023-01-11T22:18:07.2149728Z 
2023-01-11T22:18:07.2149985Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2150098Z Ran 1 test in 6.092s
2023-01-11T22:18:07.2150117Z 
2023-01-11T22:18:07.2150200Z OK
2023-01-11T22:18:07.2150220Z 
2023-01-11T22:18:07.2150335Z Generating XML reports...
2023-01-11T22:18:07.2150781Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220753.xml
2023-01-11T22:18:07.2151143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2151300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2151677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2152121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2152145Z 
2023-01-11T22:18:07.2152261Z Running tests...
2023-01-11T22:18:07.2152526Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2152843Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2153103Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2153322Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42596
2023-01-11T22:18:07.2153534Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42597
2023-01-11T22:18:07.2153885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2154049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2154423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2154602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2155029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2155269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2155644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2155824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2156050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2156286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2156676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2157063Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2157297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2157520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2158301Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:18:07.2159076Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:18:07.2159318Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2159550Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2159643Z ok (6.193s)
2023-01-11T22:18:07.2159663Z 
2023-01-11T22:18:07.2159926Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2160019Z Ran 1 test in 6.194s
2023-01-11T22:18:07.2160049Z 
2023-01-11T22:18:07.2160123Z OK
2023-01-11T22:18:07.2160141Z 
2023-01-11T22:18:07.2160262Z Generating XML reports...
2023-01-11T22:18:07.2160709Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220801.xml
2023-01-11T22:18:07.2161080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2161249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2161619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2161798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2161817Z 
2023-01-11T22:18:07.2161917Z Running tests...
2023-01-11T22:18:07.2162157Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2162465Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2162744Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2163538Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.671s)
2023-01-11T22:18:07.2163596Z 
2023-01-11T22:18:07.2163862Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2163969Z Ran 1 test in 1.671s
2023-01-11T22:18:07.2163987Z 
2023-01-11T22:18:07.2164089Z OK (skipped=1)
2023-01-11T22:18:07.2164108Z 
2023-01-11T22:18:07.2164224Z Generating XML reports...
2023-01-11T22:18:07.2164669Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220810.xml
2023-01-11T22:18:07.2165028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2165186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2165561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2165740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2165759Z 
2023-01-11T22:18:07.2165861Z Running tests...
2023-01-11T22:18:07.2166113Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2166420Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2166696Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2167430Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.665s)
2023-01-11T22:18:07.2167453Z 
2023-01-11T22:18:07.2167711Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2167804Z Ran 1 test in 1.665s
2023-01-11T22:18:07.2167835Z 
2023-01-11T22:18:07.2167924Z OK (skipped=1)
2023-01-11T22:18:07.2167943Z 
2023-01-11T22:18:07.2168062Z Generating XML reports...
2023-01-11T22:18:07.2168503Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220814.xml
2023-01-11T22:18:07.2168865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2169034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2169403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2169585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2169608Z 
2023-01-11T22:18:07.2169707Z Running tests...
2023-01-11T22:18:07.2169953Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2170268Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2170541Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2170754Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42783
2023-01-11T22:18:07.2170963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42784
2023-01-11T22:18:07.2171324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2171490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2171859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2172126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2172493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2172660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2173023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2173203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2173442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2173681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2174074Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2174476Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2174689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2174916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2175249Z STAGE:2023-01-11 22:08:23 42784:42784 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2175576Z STAGE:2023-01-11 22:08:23 42783:42783 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2175808Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2176042Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2176370Z STAGE:2023-01-11 22:08:24 42784:42784 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2176697Z STAGE:2023-01-11 22:08:24 42783:42783 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2177045Z STAGE:2023-01-11 22:08:24 42784:42784 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2177377Z STAGE:2023-01-11 22:08:24 42783:42783 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2178150Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:18:07.2178940Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:18:07.2179277Z STAGE:2023-01-11 22:08:24 42784:42784 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2179585Z STAGE:2023-01-11 22:08:24 42783:42783 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2179911Z STAGE:2023-01-11 22:08:24 42784:42784 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2180234Z STAGE:2023-01-11 22:08:24 42783:42783 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2180618Z STAGE:2023-01-11 22:08:24 42784:42784 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2181010Z STAGE:2023-01-11 22:08:24 42783:42783 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2181105Z ok (6.082s)
2023-01-11T22:18:07.2181125Z 
2023-01-11T22:18:07.2181378Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2181484Z Ran 1 test in 6.083s
2023-01-11T22:18:07.2181504Z 
2023-01-11T22:18:07.2181576Z OK
2023-01-11T22:18:07.2181606Z 
2023-01-11T22:18:07.2181710Z Generating XML reports...
2023-01-11T22:18:07.2182152Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220818.xml
2023-01-11T22:18:07.2182515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2182683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2183054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2183241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2183260Z 
2023-01-11T22:18:07.2183366Z Running tests...
2023-01-11T22:18:07.2183620Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2183916Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2184181Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2184396Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42905
2023-01-11T22:18:07.2184608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42906
2023-01-11T22:18:07.2184971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2185148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2185522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2185709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2186052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2186220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2186587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2186764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2187004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2187242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2187638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2188031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2188257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2188467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2188568Z ok (5.547s)
2023-01-11T22:18:07.2188588Z 
2023-01-11T22:18:07.2188843Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2188943Z Ran 1 test in 5.547s
2023-01-11T22:18:07.2188963Z 
2023-01-11T22:18:07.2189042Z OK
2023-01-11T22:18:07.2189061Z 
2023-01-11T22:18:07.2189179Z Generating XML reports...
2023-01-11T22:18:07.2189670Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220827.xml
2023-01-11T22:18:07.2190091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2190262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2190621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2190804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2190824Z 
2023-01-11T22:18:07.2190925Z Running tests...
2023-01-11T22:18:07.2191183Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2191484Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2191753Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2192752Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.642s)
2023-01-11T22:18:07.2192777Z 
2023-01-11T22:18:07.2193033Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2193143Z Ran 1 test in 1.642s
2023-01-11T22:18:07.2193162Z 
2023-01-11T22:18:07.2193250Z OK (skipped=1)
2023-01-11T22:18:07.2193281Z 
2023-01-11T22:18:07.2193387Z Generating XML reports...
2023-01-11T22:18:07.2193826Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220835.xml
2023-01-11T22:18:07.2194188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2194356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2194730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2194917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2194937Z 
2023-01-11T22:18:07.2195038Z Running tests...
2023-01-11T22:18:07.2195286Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2195584Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2195867Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2196079Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43054
2023-01-11T22:18:07.2196289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43055
2023-01-11T22:18:07.2196648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2196821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2197191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2197375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2197719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2197888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2198250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2198430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2198674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2199084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2199491Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2199876Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2200100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2200310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2201231Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:18:07.2201346Z   warnings.warn(
2023-01-11T22:18:07.2202253Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:18:07.2202354Z   warnings.warn(
2023-01-11T22:18:07.2202589Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2202808Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2202902Z ok (5.683s)
2023-01-11T22:18:07.2202922Z 
2023-01-11T22:18:07.2203181Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2203292Z Ran 1 test in 5.683s
2023-01-11T22:18:07.2203312Z 
2023-01-11T22:18:07.2203388Z OK
2023-01-11T22:18:07.2203421Z 
2023-01-11T22:18:07.2203527Z Generating XML reports...
2023-01-11T22:18:07.2203969Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220839.xml
2023-01-11T22:18:07.2204335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2204506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2204872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2205053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2205073Z 
2023-01-11T22:18:07.2205173Z Running tests...
2023-01-11T22:18:07.2205421Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2205726Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2206004Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2206740Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.640s)
2023-01-11T22:18:07.2206761Z 
2023-01-11T22:18:07.2207013Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2207119Z Ran 1 test in 1.640s
2023-01-11T22:18:07.2207139Z 
2023-01-11T22:18:07.2207238Z OK (skipped=1)
2023-01-11T22:18:07.2207257Z 
2023-01-11T22:18:07.2207375Z Generating XML reports...
2023-01-11T22:18:07.2207812Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220848.xml
2023-01-11T22:18:07.2208268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2208445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2208804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2208987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2209006Z 
2023-01-11T22:18:07.2209114Z Running tests...
2023-01-11T22:18:07.2209372Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2209681Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2209949Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2210164Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43206
2023-01-11T22:18:07.2210374Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43207
2023-01-11T22:18:07.2210734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2210904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2211274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2211458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2211813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2211982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2212351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2212541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2212771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2213011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2213407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2213803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2214030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2214255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2214587Z STAGE:2023-01-11 22:08:57 43207:43207 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2214921Z STAGE:2023-01-11 22:08:57 43206:43206 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2215159Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2215377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:18:07.2215709Z STAGE:2023-01-11 22:08:57 43207:43207 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2216038Z STAGE:2023-01-11 22:08:57 43206:43206 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2216380Z STAGE:2023-01-11 22:08:57 43207:43207 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2216724Z STAGE:2023-01-11 22:08:57 43206:43206 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2217046Z STAGE:2023-01-11 22:08:57 43206:43206 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2217487Z STAGE:2023-01-11 22:08:58 43206:43206 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2217834Z STAGE:2023-01-11 22:08:58 43206:43206 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2217932Z ok (6.294s)
2023-01-11T22:18:07.2217953Z 
2023-01-11T22:18:07.2218198Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2218305Z Ran 1 test in 6.294s
2023-01-11T22:18:07.2218324Z 
2023-01-11T22:18:07.2218410Z OK
2023-01-11T22:18:07.2218429Z 
2023-01-11T22:18:07.2218547Z Generating XML reports...
2023-01-11T22:18:07.2218992Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220852.xml
2023-01-11T22:18:07.2219357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2219527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2219908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2220081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2220115Z 
2023-01-11T22:18:07.2220204Z Running tests...
2023-01-11T22:18:07.2220464Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2220771Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2221035Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2221246Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43328
2023-01-11T22:18:07.2221462Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43329
2023-01-11T22:18:07.2221828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2222004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2222363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2222549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2222911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2223078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2223443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2223622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2223865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2224114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2224496Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2224888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2225119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2225345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2225444Z ok (5.667s)
2023-01-11T22:18:07.2225463Z 
2023-01-11T22:18:07.2225721Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2225825Z Ran 1 test in 5.667s
2023-01-11T22:18:07.2225844Z 
2023-01-11T22:18:07.2225927Z OK
2023-01-11T22:18:07.2225946Z 
2023-01-11T22:18:07.2226066Z Generating XML reports...
2023-01-11T22:18:07.2226600Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220901.xml
2023-01-11T22:18:07.2226974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2227145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2227520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2227703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2227722Z 
2023-01-11T22:18:07.2227822Z Running tests...
2023-01-11T22:18:07.2228084Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2228393Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2228650Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2228873Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43443
2023-01-11T22:18:07.2229082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43444
2023-01-11T22:18:07.2229442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2229614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2229986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2230170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2230529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2230746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2231110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2231290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2231531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2231768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2232473Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2232871Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2233097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2233320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2233420Z ok (5.585s)
2023-01-11T22:18:07.2233443Z 
2023-01-11T22:18:07.2233692Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2233797Z Ran 1 test in 5.585s
2023-01-11T22:18:07.2233817Z 
2023-01-11T22:18:07.2233906Z OK
2023-01-11T22:18:07.2233926Z 
2023-01-11T22:18:07.2234043Z Generating XML reports...
2023-01-11T22:18:07.2234494Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220909.xml
2023-01-11T22:18:07.2234857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2235029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2235401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2235573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2235692Z 
2023-01-11T22:18:07.2235837Z Running tests...
2023-01-11T22:18:07.2236104Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2236409Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2236688Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2237433Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.670s)
2023-01-11T22:18:07.2237454Z 
2023-01-11T22:18:07.2237706Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2237811Z Ran 1 test in 1.670s
2023-01-11T22:18:07.2237831Z 
2023-01-11T22:18:07.2237933Z OK (skipped=1)
2023-01-11T22:18:07.2237956Z 
2023-01-11T22:18:07.2238078Z Generating XML reports...
2023-01-11T22:18:07.2238505Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220917.xml
2023-01-11T22:18:07.2238872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2239040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2239407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2239591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2239610Z 
2023-01-11T22:18:07.2239723Z Running tests...
2023-01-11T22:18:07.2239976Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2240283Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2240534Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2241276Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.675s)
2023-01-11T22:18:07.2241312Z 
2023-01-11T22:18:07.2241552Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2241658Z Ran 1 test in 1.675s
2023-01-11T22:18:07.2241677Z 
2023-01-11T22:18:07.2241779Z OK (skipped=1)
2023-01-11T22:18:07.2241798Z 
2023-01-11T22:18:07.2241914Z Generating XML reports...
2023-01-11T22:18:07.2242354Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220921.xml
2023-01-11T22:18:07.2242720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2242890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2243262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2243435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2243471Z 
2023-01-11T22:18:07.2243561Z Running tests...
2023-01-11T22:18:07.2243820Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2244128Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2244415Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2245199Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.667s)
2023-01-11T22:18:07.2245258Z 
2023-01-11T22:18:07.2245520Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2245625Z Ran 1 test in 1.667s
2023-01-11T22:18:07.2245644Z 
2023-01-11T22:18:07.2245741Z OK (skipped=1)
2023-01-11T22:18:07.2245760Z 
2023-01-11T22:18:07.2245879Z Generating XML reports...
2023-01-11T22:18:07.2246311Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220925.xml
2023-01-11T22:18:07.2246678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2246855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2247230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2247424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2247443Z 
2023-01-11T22:18:07.2247545Z Running tests...
2023-01-11T22:18:07.2247804Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2248112Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2248392Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2248607Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43660
2023-01-11T22:18:07.2248810Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43661
2023-01-11T22:18:07.2249171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2249339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2249713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2249897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2250256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2250420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2250769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2250953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2251190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2251425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2251823Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2252214Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2252442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2252668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2252758Z ok (6.161s)
2023-01-11T22:18:07.2252778Z 
2023-01-11T22:18:07.2253026Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2253127Z Ran 1 test in 6.161s
2023-01-11T22:18:07.2253147Z 
2023-01-11T22:18:07.2253235Z OK
2023-01-11T22:18:07.2253254Z 
2023-01-11T22:18:07.2253372Z Generating XML reports...
2023-01-11T22:18:07.2253811Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220930.xml
2023-01-11T22:18:07.2254271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2254441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2254812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2254982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2255015Z 
2023-01-11T22:18:07.2255103Z Running tests...
2023-01-11T22:18:07.2255358Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2255660Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2255923Z   test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2256135Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43779
2023-01-11T22:18:07.2256357Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43780
2023-01-11T22:18:07.2256717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2256884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2257240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2257423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2257773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2257941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2258309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2258497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2258735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2258975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2259354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2259741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2259967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2260339Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2023-01-11T22:18:07.2260583Z   warnings.warn("Initializing zero-element tensors is a no-op")
2023-01-11T22:18:07.2260809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2261177Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2023-01-11T22:18:07.2261425Z   warnings.warn("Initializing zero-element tensors is a no-op")
2023-01-11T22:18:07.2261523Z ok (5.678s)
2023-01-11T22:18:07.2261542Z 
2023-01-11T22:18:07.2261784Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2261891Z Ran 1 test in 5.678s
2023-01-11T22:18:07.2261910Z 
2023-01-11T22:18:07.2261993Z OK
2023-01-11T22:18:07.2262012Z 
2023-01-11T22:18:07.2262127Z Generating XML reports...
2023-01-11T22:18:07.2262566Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220938.xml
2023-01-11T22:18:07.2262925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2263189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2263562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2263734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2263768Z 
2023-01-11T22:18:07.2263858Z Running tests...
2023-01-11T22:18:07.2264112Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2264421Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2264680Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2264895Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43894
2023-01-11T22:18:07.2265107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43895
2023-01-11T22:18:07.2265474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2265645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2266005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2266188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2266540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2266701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2267069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2267252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2267499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2267740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2268120Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2268511Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2268736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2268971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:18:07.2269191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2269419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:18:07.2269817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.2270202Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.2270300Z ok (4.398s)
2023-01-11T22:18:07.2270321Z 
2023-01-11T22:18:07.2270562Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2270671Z Ran 1 test in 4.399s
2023-01-11T22:18:07.2270690Z 
2023-01-11T22:18:07.2270780Z OK
2023-01-11T22:18:07.2270800Z 
2023-01-11T22:18:07.2270921Z Generating XML reports...
2023-01-11T22:18:07.2271366Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220946.xml
2023-01-11T22:18:07.2271734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2272135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2272682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2272881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2272901Z 
2023-01-11T22:18:07.2272990Z Running tests...
2023-01-11T22:18:07.2273253Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2273566Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2273821Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2274040Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44001
2023-01-11T22:18:07.2274256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44002
2023-01-11T22:18:07.2274625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2274806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2275166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2275359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2275720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2275892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2276261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2276448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2276695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2276944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2277341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2277717Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2277947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2278188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:18:07.2278411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2278653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:18:07.2279048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.2279450Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.2279552Z ok (4.388s)
2023-01-11T22:18:07.2279573Z 
2023-01-11T22:18:07.2279836Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2279929Z Ran 1 test in 4.388s
2023-01-11T22:18:07.2279948Z 
2023-01-11T22:18:07.2280038Z OK
2023-01-11T22:18:07.2280058Z 
2023-01-11T22:18:07.2280181Z Generating XML reports...
2023-01-11T22:18:07.2280629Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220953.xml
2023-01-11T22:18:07.2280998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2281171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2281598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2281854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2281873Z 
2023-01-11T22:18:07.2281981Z Running tests...
2023-01-11T22:18:07.2282223Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2282534Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2282816Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2283561Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.668s)
2023-01-11T22:18:07.2283581Z 
2023-01-11T22:18:07.2283843Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2283956Z Ran 1 test in 1.668s
2023-01-11T22:18:07.2283976Z 
2023-01-11T22:18:07.2284083Z OK (skipped=1)
2023-01-11T22:18:07.2284102Z 
2023-01-11T22:18:07.2284225Z Generating XML reports...
2023-01-11T22:18:07.2284673Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221000.xml
2023-01-11T22:18:07.2285025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2285201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2285578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2285766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2285785Z 
2023-01-11T22:18:07.2285895Z Running tests...
2023-01-11T22:18:07.2286162Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2286476Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2286757Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2287495Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78748 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.674s)
2023-01-11T22:18:07.2287516Z 
2023-01-11T22:18:07.2287773Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2287866Z Ran 1 test in 1.675s
2023-01-11T22:18:07.2287885Z 
2023-01-11T22:18:07.2287991Z OK (skipped=1)
2023-01-11T22:18:07.2288009Z 
2023-01-11T22:18:07.2288132Z Generating XML reports...
2023-01-11T22:18:07.2288583Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221005.xml
2023-01-11T22:18:07.2288951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2289125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2289502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2289690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2289710Z 
2023-01-11T22:18:07.2289816Z Running tests...
2023-01-11T22:18:07.2290058Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2290368Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2290642Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2290958Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44176
2023-01-11T22:18:07.2291181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44177
2023-01-11T22:18:07.2291551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2291724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2292099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2292269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2292633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2292804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2293178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2293360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2293606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2293851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2294252Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2294649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2294861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2295091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2295197Z ok (4.273s)
2023-01-11T22:18:07.2295217Z 
2023-01-11T22:18:07.2295483Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2295593Z Ran 1 test in 4.273s
2023-01-11T22:18:07.2295612Z 
2023-01-11T22:18:07.2295704Z OK
2023-01-11T22:18:07.2295723Z 
2023-01-11T22:18:07.2295845Z Generating XML reports...
2023-01-11T22:18:07.2296296Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221009.xml
2023-01-11T22:18:07.2296645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2296821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2297196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2297386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2297409Z 
2023-01-11T22:18:07.2297515Z Running tests...
2023-01-11T22:18:07.2297778Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2298091Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2298341Z   test_gather (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.2298362Z 
2023-01-11T22:18:07.2298619Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2298712Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2298731Z 
2023-01-11T22:18:07.2298838Z OK (skipped=1)
2023-01-11T22:18:07.2298857Z 
2023-01-11T22:18:07.2298979Z Generating XML reports...
2023-01-11T22:18:07.2299424Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221016.xml
2023-01-11T22:18:07.2299791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2300063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2300452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2300639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2300659Z 
2023-01-11T22:18:07.2300766Z Running tests...
2023-01-11T22:18:07.2301010Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2301321Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2301580Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.2301599Z 
2023-01-11T22:18:07.2301857Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2301968Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2301990Z 
2023-01-11T22:18:07.2302097Z OK (skipped=1)
2023-01-11T22:18:07.2302116Z 
2023-01-11T22:18:07.2302240Z Generating XML reports...
2023-01-11T22:18:07.2302690Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221018.xml
2023-01-11T22:18:07.2303039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2303211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2303588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2303777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2303796Z 
2023-01-11T22:18:07.2303902Z Running tests...
2023-01-11T22:18:07.2304161Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2304470Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2304731Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2304951Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44347
2023-01-11T22:18:07.2305154Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44348
2023-01-11T22:18:07.2305519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2305692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2306067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2306254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2306613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2306790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2307155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2307324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2307567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2307812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2308211Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2308607Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2308837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2309163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2309513Z STAGE:2023-01-11 22:10:25 44348:44348 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2309838Z STAGE:2023-01-11 22:10:25 44347:44347 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2310156Z STAGE:2023-01-11 22:10:26 44347:44347 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2310500Z STAGE:2023-01-11 22:10:26 44347:44347 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2310834Z STAGE:2023-01-11 22:10:26 44348:44348 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2311178Z STAGE:2023-01-11 22:10:26 44348:44348 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2311507Z STAGE:2023-01-11 22:10:26 44347:44347 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2312065Z STAGE:2023-01-11 22:10:26 44348:44348 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2312431Z STAGE:2023-01-11 22:10:26 44348:44348 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2312780Z STAGE:2023-01-11 22:10:26 44348:44348 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2313117Z STAGE:2023-01-11 22:10:26 44347:44347 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2313444Z STAGE:2023-01-11 22:10:26 44347:44347 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2313546Z ok (6.364s)
2023-01-11T22:18:07.2313566Z 
2023-01-11T22:18:07.2313829Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2313937Z Ran 1 test in 6.364s
2023-01-11T22:18:07.2313956Z 
2023-01-11T22:18:07.2314047Z OK
2023-01-11T22:18:07.2314067Z 
2023-01-11T22:18:07.2314191Z Generating XML reports...
2023-01-11T22:18:07.2314649Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221020.xml
2023-01-11T22:18:07.2315017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2315192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2315554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2315743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2315762Z 
2023-01-11T22:18:07.2315869Z Running tests...
2023-01-11T22:18:07.2316131Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2316445Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2316713Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.2316735Z 
2023-01-11T22:18:07.2316994Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2317107Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2317127Z 
2023-01-11T22:18:07.2317215Z OK (skipped=1)
2023-01-11T22:18:07.2317252Z 
2023-01-11T22:18:07.2317355Z Generating XML reports...
2023-01-11T22:18:07.2317803Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221029.xml
2023-01-11T22:18:07.2318171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2318345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2318724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2318916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2319016Z 
2023-01-11T22:18:07.2319179Z Running tests...
2023-01-11T22:18:07.2319448Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2319747Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2320006Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.2320026Z 
2023-01-11T22:18:07.2320285Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2320396Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2320414Z 
2023-01-11T22:18:07.2320521Z OK (skipped=1)
2023-01-11T22:18:07.2320539Z 
2023-01-11T22:18:07.2320662Z Generating XML reports...
2023-01-11T22:18:07.2321110Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221032.xml
2023-01-11T22:18:07.2321483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2321661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2322019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2322207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2322226Z 
2023-01-11T22:18:07.2322333Z Running tests...
2023-01-11T22:18:07.2322591Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2322900Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2323156Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2323378Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44536
2023-01-11T22:18:07.2323594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44537
2023-01-11T22:18:07.2323953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2324128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2324506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2324694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2341837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2342065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2342492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2342674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2342925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2343156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2343551Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2343937Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2344157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2344374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2344463Z ok (5.792s)
2023-01-11T22:18:07.2344485Z 
2023-01-11T22:18:07.2344743Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2344843Z Ran 1 test in 5.793s
2023-01-11T22:18:07.2344963Z 
2023-01-11T22:18:07.2345048Z OK
2023-01-11T22:18:07.2345111Z 
2023-01-11T22:18:07.2345223Z Generating XML reports...
2023-01-11T22:18:07.2345674Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221034.xml
2023-01-11T22:18:07.2346034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2346195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2346562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2346740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2346759Z 
2023-01-11T22:18:07.2346854Z Running tests...
2023-01-11T22:18:07.2347104Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2347406Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2347669Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2348410Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.633s)
2023-01-11T22:18:07.2348431Z 
2023-01-11T22:18:07.2348679Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2348778Z Ran 1 test in 1.634s
2023-01-11T22:18:07.2348797Z 
2023-01-11T22:18:07.2348892Z OK (skipped=1)
2023-01-11T22:18:07.2348910Z 
2023-01-11T22:18:07.2349022Z Generating XML reports...
2023-01-11T22:18:07.2349461Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221042.xml
2023-01-11T22:18:07.2349827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2349988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2350347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2350527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2350546Z 
2023-01-11T22:18:07.2350641Z Running tests...
2023-01-11T22:18:07.2350884Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2351185Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2351425Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2351633Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44688
2023-01-11T22:18:07.2352177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44689
2023-01-11T22:18:07.2352573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2352730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2353100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2353278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2353626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2353787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2354150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2354441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2354728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2354962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2355355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2355741Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2355957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2356187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:18:07.2356398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2356630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:18:07.2357016Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.2357397Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.2357481Z ok (4.403s)
2023-01-11T22:18:07.2357508Z 
2023-01-11T22:18:07.2357755Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2357852Z Ran 1 test in 4.403s
2023-01-11T22:18:07.2357872Z 
2023-01-11T22:18:07.2357951Z OK
2023-01-11T22:18:07.2357970Z 
2023-01-11T22:18:07.2358081Z Generating XML reports...
2023-01-11T22:18:07.2358524Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221046.xml
2023-01-11T22:18:07.2358894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2359070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2359448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2359620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2359639Z 
2023-01-11T22:18:07.2359743Z Running tests...
2023-01-11T22:18:07.2360003Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2360318Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2360566Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2360787Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44795
2023-01-11T22:18:07.2361006Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44796
2023-01-11T22:18:07.2361377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2361534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2361909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2362098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2362459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2362631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2363007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2363192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2363596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2363846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2364231Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2364623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2364854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2365082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2365185Z ok (5.649s)
2023-01-11T22:18:07.2365205Z 
2023-01-11T22:18:07.2365468Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2365579Z Ran 1 test in 5.649s
2023-01-11T22:18:07.2365602Z 
2023-01-11T22:18:07.2365694Z OK
2023-01-11T22:18:07.2365714Z 
2023-01-11T22:18:07.2365840Z Generating XML reports...
2023-01-11T22:18:07.2366271Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221053.xml
2023-01-11T22:18:07.2366640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2366815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2367191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2367380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2367400Z 
2023-01-11T22:18:07.2367504Z Running tests...
2023-01-11T22:18:07.2367763Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2368075Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2368311Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2368530Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44910
2023-01-11T22:18:07.2368749Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44911
2023-01-11T22:18:07.2369115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2369289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2369660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2369849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2370212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2370393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2370747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2370933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2371179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2371426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2371823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2372222Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2372451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2372771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2372858Z ok (4.445s)
2023-01-11T22:18:07.2372895Z 
2023-01-11T22:18:07.2373143Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2373254Z Ran 1 test in 4.445s
2023-01-11T22:18:07.2373274Z 
2023-01-11T22:18:07.2373363Z OK
2023-01-11T22:18:07.2373383Z 
2023-01-11T22:18:07.2373505Z Generating XML reports...
2023-01-11T22:18:07.2373954Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221102.xml
2023-01-11T22:18:07.2374323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2374496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2374874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2375052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2375071Z 
2023-01-11T22:18:07.2375181Z Running tests...
2023-01-11T22:18:07.2375442Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2375755Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2376024Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2376246Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45015
2023-01-11T22:18:07.2376461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45016
2023-01-11T22:18:07.2376825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2376980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2377361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2377555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2377917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2378085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2378452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2378633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2378879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2379126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2379510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2379912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2380141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2380380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:18:07.2380609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2380852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:18:07.2381249Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.2381644Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.2381800Z ok (4.391s)
2023-01-11T22:18:07.2381857Z 
2023-01-11T22:18:07.2382105Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2382217Z Ran 1 test in 4.391s
2023-01-11T22:18:07.2382237Z 
2023-01-11T22:18:07.2382330Z OK
2023-01-11T22:18:07.2382349Z 
2023-01-11T22:18:07.2382473Z Generating XML reports...
2023-01-11T22:18:07.2382919Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221108.xml
2023-01-11T22:18:07.2383287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2383461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2383840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2384011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2384052Z 
2023-01-11T22:18:07.2384145Z Running tests...
2023-01-11T22:18:07.2384400Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2384711Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2384968Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2385184Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45122
2023-01-11T22:18:07.2385399Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45123
2023-01-11T22:18:07.2385765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2385936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2386296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2386492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2386855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2387027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2387399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2387582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2387826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2388070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2388449Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2388848Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2389079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2389320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:18:07.2389542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2389775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:18:07.2390172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.2390563Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.2390662Z ok (4.321s)
2023-01-11T22:18:07.2390732Z 
2023-01-11T22:18:07.2391038Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2391135Z Ran 1 test in 4.321s
2023-01-11T22:18:07.2391155Z 
2023-01-11T22:18:07.2391245Z OK
2023-01-11T22:18:07.2391265Z 
2023-01-11T22:18:07.2391388Z Generating XML reports...
2023-01-11T22:18:07.2392061Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221115.xml
2023-01-11T22:18:07.2392467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2392642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2393022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2393212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2393232Z 
2023-01-11T22:18:07.2393321Z Running tests...
2023-01-11T22:18:07.2393592Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2393907Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2394171Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2394389Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45229
2023-01-11T22:18:07.2394605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45230
2023-01-11T22:18:07.2394971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2395144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2395502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2395691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2396059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2396229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2396602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2396788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2397033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2397278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2397673Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2398049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2398286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2398514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2398615Z ok (5.579s)
2023-01-11T22:18:07.2398635Z 
2023-01-11T22:18:07.2398895Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2399005Z Ran 1 test in 5.579s
2023-01-11T22:18:07.2399024Z 
2023-01-11T22:18:07.2399113Z OK
2023-01-11T22:18:07.2399133Z 
2023-01-11T22:18:07.2399255Z Generating XML reports...
2023-01-11T22:18:07.2399704Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221122.xml
2023-01-11T22:18:07.2400053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2400225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2400742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2400937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2400956Z 
2023-01-11T22:18:07.2401063Z Running tests...
2023-01-11T22:18:07.2401320Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2401633Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2401872Z   test_irecv (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support irecv (0.002s)
2023-01-11T22:18:07.2401891Z 
2023-01-11T22:18:07.2402130Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2402241Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2402260Z 
2023-01-11T22:18:07.2402366Z OK (skipped=1)
2023-01-11T22:18:07.2402384Z 
2023-01-11T22:18:07.2402510Z Generating XML reports...
2023-01-11T22:18:07.2402962Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221130.xml
2023-01-11T22:18:07.2403329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2403502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2403877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2404064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2404083Z 
2023-01-11T22:18:07.2404172Z Running tests...
2023-01-11T22:18:07.2404432Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2404745Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2404976Z   test_isend (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2023-01-11T22:18:07.2405002Z 
2023-01-11T22:18:07.2405259Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2405368Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2405387Z 
2023-01-11T22:18:07.2405490Z OK (skipped=1)
2023-01-11T22:18:07.2405509Z 
2023-01-11T22:18:07.2405633Z Generating XML reports...
2023-01-11T22:18:07.2406080Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221133.xml
2023-01-11T22:18:07.2406430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2406604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2406980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2407169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2407192Z 
2023-01-11T22:18:07.2407298Z Running tests...
2023-01-11T22:18:07.2407557Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2407869Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2408131Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2023-01-11T22:18:07.2408151Z 
2023-01-11T22:18:07.2408407Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2408500Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2408519Z 
2023-01-11T22:18:07.2408626Z OK (skipped=1)
2023-01-11T22:18:07.2408644Z 
2023-01-11T22:18:07.2408763Z Generating XML reports...
2023-01-11T22:18:07.2409209Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221135.xml
2023-01-11T22:18:07.2409621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2409840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2410216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2410404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2410423Z 
2023-01-11T22:18:07.2410511Z Running tests...
2023-01-11T22:18:07.2410768Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2411081Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2411340Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support isend (0.002s)
2023-01-11T22:18:07.2411359Z 
2023-01-11T22:18:07.2411615Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2411726Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2411745Z 
2023-01-11T22:18:07.2411854Z OK (skipped=1)
2023-01-11T22:18:07.2411873Z 
2023-01-11T22:18:07.2411994Z Generating XML reports...
2023-01-11T22:18:07.2412440Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221138.xml
2023-01-11T22:18:07.2412790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2412962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2413337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2413525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2413544Z 
2023-01-11T22:18:07.2413651Z Running tests...
2023-01-11T22:18:07.2413912Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2414226Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2414511Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2414713Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45479
2023-01-11T22:18:07.2414930Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45480
2023-01-11T22:18:07.2415296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2415468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2415840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2416029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2416388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2416567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2416938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2417106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2417352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2417596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2417995Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2418391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2418618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2418957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:18:07.2419187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2419423Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:18:07.2419804Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.2420198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.2420438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:18:07.2420680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:18:07.2421081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:18:07.2421470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:18:07.2421701Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2023-01-11T22:18:07.2421801Z ok (21.734s)
2023-01-11T22:18:07.2421821Z 
2023-01-11T22:18:07.2422082Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2422176Z Ran 1 test in 21.734s
2023-01-11T22:18:07.2422195Z 
2023-01-11T22:18:07.2422284Z OK
2023-01-11T22:18:07.2422303Z 
2023-01-11T22:18:07.2422424Z Generating XML reports...
2023-01-11T22:18:07.2422876Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221140.xml
2023-01-11T22:18:07.2423242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2423422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2423802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2423991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2424011Z 
2023-01-11T22:18:07.2424116Z Running tests...
2023-01-11T22:18:07.2424360Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2424671Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2424975Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2425195Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45602
2023-01-11T22:18:07.2425411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45603
2023-01-11T22:18:07.2425783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2425957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2426336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2426506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2426867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2427038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2427405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2427589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2427937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2428187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2428586Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2428981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2429192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2429434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:18:07.2429661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2429895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:18:07.2430297Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.2430687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.2430968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:18:07.2431210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:18:07.2431604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:18:07.2432202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:18:07.2432446Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2023-01-11T22:18:07.2432684Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2023-01-11T22:18:07.2432786Z ok (21.521s)
2023-01-11T22:18:07.2432807Z 
2023-01-11T22:18:07.2433078Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2433192Z Ran 1 test in 21.521s
2023-01-11T22:18:07.2433212Z 
2023-01-11T22:18:07.2433303Z OK
2023-01-11T22:18:07.2433322Z 
2023-01-11T22:18:07.2433443Z Generating XML reports...
2023-01-11T22:18:07.2433878Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221204.xml
2023-01-11T22:18:07.2434249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2434424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2434799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2434992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2435012Z 
2023-01-11T22:18:07.2435122Z Running tests...
2023-01-11T22:18:07.2435382Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2435693Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2436105Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:18:07.2436126Z 
2023-01-11T22:18:07.2436367Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2436478Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2436497Z 
2023-01-11T22:18:07.2436603Z OK (skipped=1)
2023-01-11T22:18:07.2436622Z 
2023-01-11T22:18:07.2436744Z Generating XML reports...
2023-01-11T22:18:07.2437184Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221228.xml
2023-01-11T22:18:07.2437699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2437876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2438252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2438437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2438457Z 
2023-01-11T22:18:07.2438545Z Running tests...
2023-01-11T22:18:07.2438805Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2439115Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2439504Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:18:07.2439526Z 
2023-01-11T22:18:07.2439786Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2439897Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2439917Z 
2023-01-11T22:18:07.2440020Z OK (skipped=1)
2023-01-11T22:18:07.2440039Z 
2023-01-11T22:18:07.2440160Z Generating XML reports...
2023-01-11T22:18:07.2440603Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221231.xml
2023-01-11T22:18:07.2440951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2441126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2441500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2441689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2441707Z 
2023-01-11T22:18:07.2441813Z Running tests...
2023-01-11T22:18:07.2442080Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2442393Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2442813Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:18:07.2442833Z 
2023-01-11T22:18:07.2443087Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2443178Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2443197Z 
2023-01-11T22:18:07.2443304Z OK (skipped=1)
2023-01-11T22:18:07.2443323Z 
2023-01-11T22:18:07.2443445Z Generating XML reports...
2023-01-11T22:18:07.2443887Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221233.xml
2023-01-11T22:18:07.2444254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2444435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2444813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2445004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2445023Z 
2023-01-11T22:18:07.2445113Z Running tests...
2023-01-11T22:18:07.2445371Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2445682Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2446094Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:18:07.2446114Z 
2023-01-11T22:18:07.2446368Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2446478Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2446547Z 
2023-01-11T22:18:07.2446660Z OK (skipped=1)
2023-01-11T22:18:07.2446717Z 
2023-01-11T22:18:07.2446845Z Generating XML reports...
2023-01-11T22:18:07.2447290Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221236.xml
2023-01-11T22:18:07.2447636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2447810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2448184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2448369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2448388Z 
2023-01-11T22:18:07.2448493Z Running tests...
2023-01-11T22:18:07.2448751Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2449062Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2449479Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:18:07.2449499Z 
2023-01-11T22:18:07.2449756Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2449848Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2449867Z 
2023-01-11T22:18:07.2449973Z OK (skipped=1)
2023-01-11T22:18:07.2449992Z 
2023-01-11T22:18:07.2450113Z Generating XML reports...
2023-01-11T22:18:07.2450553Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221238.xml
2023-01-11T22:18:07.2450919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2451092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2451473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2451667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2451687Z 
2023-01-11T22:18:07.2451795Z Running tests...
2023-01-11T22:18:07.2452035Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2452350Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2452625Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2452848Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45890
2023-01-11T22:18:07.2453067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45891
2023-01-11T22:18:07.2453432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2453603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2453988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2454160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2454522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2454692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2455060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2455246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2455498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2455739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2456233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2456636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2456847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2457071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2457173Z ok (5.194s)
2023-01-11T22:18:07.2457193Z 
2023-01-11T22:18:07.2457451Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2457560Z Ran 1 test in 5.194s
2023-01-11T22:18:07.2457580Z 
2023-01-11T22:18:07.2457671Z OK
2023-01-11T22:18:07.2457690Z 
2023-01-11T22:18:07.2457808Z Generating XML reports...
2023-01-11T22:18:07.2458253Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221240.xml
2023-01-11T22:18:07.2458612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2458788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2459166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2459355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2459374Z 
2023-01-11T22:18:07.2459478Z Running tests...
2023-01-11T22:18:07.2459734Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2460048Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2460323Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2460547Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46004
2023-01-11T22:18:07.2460747Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46005
2023-01-11T22:18:07.2461117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2461290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2461664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2461851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2462209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2462382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2462748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2462925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2463170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2463414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2463811Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2464200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2464428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2464651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2464750Z ok (5.202s)
2023-01-11T22:18:07.2464770Z 
2023-01-11T22:18:07.2465125Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2465224Z Ran 1 test in 5.202s
2023-01-11T22:18:07.2465244Z 
2023-01-11T22:18:07.2465334Z OK
2023-01-11T22:18:07.2465353Z 
2023-01-11T22:18:07.2465474Z Generating XML reports...
2023-01-11T22:18:07.2465921Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221248.xml
2023-01-11T22:18:07.2466287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2466457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2466835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2467024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2467043Z 
2023-01-11T22:18:07.2467151Z Running tests...
2023-01-11T22:18:07.2467404Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2467719Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2467993Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2468210Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46118
2023-01-11T22:18:07.2468428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46119
2023-01-11T22:18:07.2468797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2468968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2469342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2469514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2469887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2470057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2470427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2470615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2470863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2471108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2471504Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2472121Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2472355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2472587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2472689Z ok (5.648s)
2023-01-11T22:18:07.2472710Z 
2023-01-11T22:18:07.2472984Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2473093Z Ran 1 test in 5.649s
2023-01-11T22:18:07.2473113Z 
2023-01-11T22:18:07.2473202Z OK
2023-01-11T22:18:07.2473221Z 
2023-01-11T22:18:07.2473341Z Generating XML reports...
2023-01-11T22:18:07.2473791Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221256.xml
2023-01-11T22:18:07.2474141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2474311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2474828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2475024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2475044Z 
2023-01-11T22:18:07.2475151Z Running tests...
2023-01-11T22:18:07.2475418Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2475728Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2476002Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2476222Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46233
2023-01-11T22:18:07.2476422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46234
2023-01-11T22:18:07.2476787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2476965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2477344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2477529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2477886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2478058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2478429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2478598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2478838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2479088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2479489Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2479884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2480111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2480340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2480438Z ok (5.238s)
2023-01-11T22:18:07.2480457Z 
2023-01-11T22:18:07.2480715Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2480807Z Ran 1 test in 5.238s
2023-01-11T22:18:07.2480826Z 
2023-01-11T22:18:07.2480916Z OK
2023-01-11T22:18:07.2480935Z 
2023-01-11T22:18:07.2481057Z Generating XML reports...
2023-01-11T22:18:07.2481509Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221304.xml
2023-01-11T22:18:07.2481879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2482051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2482429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2482619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2482639Z 
2023-01-11T22:18:07.2482746Z Running tests...
2023-01-11T22:18:07.2482991Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2483302Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2483621Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2484418Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81022 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.673s)
2023-01-11T22:18:07.2484439Z 
2023-01-11T22:18:07.2484699Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2484806Z Ran 1 test in 1.673s
2023-01-11T22:18:07.2484825Z 
2023-01-11T22:18:07.2484930Z OK (skipped=1)
2023-01-11T22:18:07.2484948Z 
2023-01-11T22:18:07.2485073Z Generating XML reports...
2023-01-11T22:18:07.2485522Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221312.xml
2023-01-11T22:18:07.2485873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2486055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2486435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2486624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2486643Z 
2023-01-11T22:18:07.2486750Z Running tests...
2023-01-11T22:18:07.2487006Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2487316Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2487563Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T22:18:07.2487583Z 
2023-01-11T22:18:07.2487839Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2487931Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2487951Z 
2023-01-11T22:18:07.2488058Z OK (skipped=1)
2023-01-11T22:18:07.2488077Z 
2023-01-11T22:18:07.2488203Z Generating XML reports...
2023-01-11T22:18:07.2488650Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221316.xml
2023-01-11T22:18:07.2489015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2489187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2489565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2489753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2489773Z 
2023-01-11T22:18:07.2489879Z Running tests...
2023-01-11T22:18:07.2490120Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2490430Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2490707Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T22:18:07.2490727Z 
2023-01-11T22:18:07.2490983Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2491091Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2491110Z 
2023-01-11T22:18:07.2491212Z OK (skipped=1)
2023-01-11T22:18:07.2491231Z 
2023-01-11T22:18:07.2491352Z Generating XML reports...
2023-01-11T22:18:07.2491794Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221318.xml
2023-01-11T22:18:07.2492143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2492316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2492692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2492987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2493008Z 
2023-01-11T22:18:07.2493119Z Running tests...
2023-01-11T22:18:07.2493380Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2493694Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2494006Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T22:18:07.2494026Z 
2023-01-11T22:18:07.2494277Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2494368Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2494405Z 
2023-01-11T22:18:07.2494493Z OK (skipped=1)
2023-01-11T22:18:07.2494511Z 
2023-01-11T22:18:07.2494630Z Generating XML reports...
2023-01-11T22:18:07.2495080Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221321.xml
2023-01-11T22:18:07.2495451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2495621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2495994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2496181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2496200Z 
2023-01-11T22:18:07.2496305Z Running tests...
2023-01-11T22:18:07.2496546Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2496858Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2497163Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2497387Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46480
2023-01-11T22:18:07.2497604Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46481
2023-01-11T22:18:07.2497974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2498148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2498527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2498698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2499057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2499227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2499600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2499789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2500035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2500281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2500678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2501074Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2501285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2501514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2501613Z ok (4.252s)
2023-01-11T22:18:07.2501687Z 
2023-01-11T22:18:07.2502034Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2502150Z Ran 1 test in 4.252s
2023-01-11T22:18:07.2502170Z 
2023-01-11T22:18:07.2502258Z OK
2023-01-11T22:18:07.2502277Z 
2023-01-11T22:18:07.2502400Z Generating XML reports...
2023-01-11T22:18:07.2502850Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221323.xml
2023-01-11T22:18:07.2503220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2503377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2503751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2503938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2503957Z 
2023-01-11T22:18:07.2504066Z Running tests...
2023-01-11T22:18:07.2504328Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2504639Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2504936Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2505159Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46585
2023-01-11T22:18:07.2505359Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46586
2023-01-11T22:18:07.2505724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2505898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2506270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2506463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2506822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2506995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2507360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2507544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2507774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2508015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2508411Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2508810Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2509043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2509273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2509375Z ok (4.262s)
2023-01-11T22:18:07.2509395Z 
2023-01-11T22:18:07.2509657Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2509749Z Ran 1 test in 4.262s
2023-01-11T22:18:07.2509786Z 
2023-01-11T22:18:07.2509859Z OK
2023-01-11T22:18:07.2509877Z 
2023-01-11T22:18:07.2510000Z Generating XML reports...
2023-01-11T22:18:07.2510449Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221330.xml
2023-01-11T22:18:07.2510818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2511091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2511477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2511667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2511686Z 
2023-01-11T22:18:07.2511794Z Running tests...
2023-01-11T22:18:07.2512275Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2512595Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2512873Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T22:18:07.2512893Z 
2023-01-11T22:18:07.2513152Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2513262Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2513281Z 
2023-01-11T22:18:07.2513391Z OK (skipped=1)
2023-01-11T22:18:07.2513410Z 
2023-01-11T22:18:07.2513535Z Generating XML reports...
2023-01-11T22:18:07.2513980Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221337.xml
2023-01-11T22:18:07.2514347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2514501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2514876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2515065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2515084Z 
2023-01-11T22:18:07.2515188Z Running tests...
2023-01-11T22:18:07.2515444Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2515753Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2516061Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T22:18:07.2516081Z 
2023-01-11T22:18:07.2516339Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2516451Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2516471Z 
2023-01-11T22:18:07.2516559Z OK (skipped=1)
2023-01-11T22:18:07.2516578Z 
2023-01-11T22:18:07.2516700Z Generating XML reports...
2023-01-11T22:18:07.2517145Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221339.xml
2023-01-11T22:18:07.2517510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2517681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2518058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2518253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2518273Z 
2023-01-11T22:18:07.2518378Z Running tests...
2023-01-11T22:18:07.2518618Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2518929Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2519205Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2519948Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.686s)
2023-01-11T22:18:07.2519968Z 
2023-01-11T22:18:07.2520222Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2520464Z Ran 1 test in 1.686s
2023-01-11T22:18:07.2520485Z 
2023-01-11T22:18:07.2520594Z OK (skipped=1)
2023-01-11T22:18:07.2520613Z 
2023-01-11T22:18:07.2520735Z Generating XML reports...
2023-01-11T22:18:07.2521184Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221342.xml
2023-01-11T22:18:07.2521552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2521708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2522086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2522275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2522295Z 
2023-01-11T22:18:07.2522399Z Running tests...
2023-01-11T22:18:07.2522655Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2522971Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2523249Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2523470Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46790
2023-01-11T22:18:07.2523668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46791
2023-01-11T22:18:07.2524039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2524211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2524585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2524774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2525138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2525308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2525678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2525864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2526093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2526338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2526739Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2527134Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2527367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2527595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2527694Z ok (6.150s)
2023-01-11T22:18:07.2527714Z 
2023-01-11T22:18:07.2527975Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2528085Z Ran 1 test in 6.150s
2023-01-11T22:18:07.2528105Z 
2023-01-11T22:18:07.2528178Z OK
2023-01-11T22:18:07.2528197Z 
2023-01-11T22:18:07.2528318Z Generating XML reports...
2023-01-11T22:18:07.2528769Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221346.xml
2023-01-11T22:18:07.2529136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2529309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2529781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2529973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2529993Z 
2023-01-11T22:18:07.2530102Z Running tests...
2023-01-11T22:18:07.2530345Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2530657Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2530976Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2531199Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46909
2023-01-11T22:18:07.2531419Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46910
2023-01-11T22:18:07.2531788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2531968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2532344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2532532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2532873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2533046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2533415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2533599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2533846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2534093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2534492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2534884Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2535112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2535323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2535423Z ok (5.580s)
2023-01-11T22:18:07.2535443Z 
2023-01-11T22:18:07.2535706Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2535816Z Ran 1 test in 5.580s
2023-01-11T22:18:07.2535836Z 
2023-01-11T22:18:07.2535928Z OK
2023-01-11T22:18:07.2535947Z 
2023-01-11T22:18:07.2536072Z Generating XML reports...
2023-01-11T22:18:07.2536528Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221355.xml
2023-01-11T22:18:07.2536900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2537056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2537433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2537621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2537641Z 
2023-01-11T22:18:07.2537749Z Running tests...
2023-01-11T22:18:07.2538007Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2538320Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2538611Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2538926Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47024
2023-01-11T22:18:07.2539129Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47025
2023-01-11T22:18:07.2539500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2539673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2540048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2540233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2540592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2540764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2541140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2541326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2541554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2541795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2542197Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2542592Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2542821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2543042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2543148Z ok (5.791s)
2023-01-11T22:18:07.2543170Z 
2023-01-11T22:18:07.2543431Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2543541Z Ran 1 test in 5.791s
2023-01-11T22:18:07.2543561Z 
2023-01-11T22:18:07.2543633Z OK
2023-01-11T22:18:07.2543652Z 
2023-01-11T22:18:07.2543774Z Generating XML reports...
2023-01-11T22:18:07.2544219Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221403.xml
2023-01-11T22:18:07.2544581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2544751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2545127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2545316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2545339Z 
2023-01-11T22:18:07.2545449Z Running tests...
2023-01-11T22:18:07.2545691Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2545998Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2546277Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2547020Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.642s)
2023-01-11T22:18:07.2547041Z 
2023-01-11T22:18:07.2547298Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2547406Z Ran 1 test in 1.642s
2023-01-11T22:18:07.2547425Z 
2023-01-11T22:18:07.2547533Z OK (skipped=1)
2023-01-11T22:18:07.2547599Z 
2023-01-11T22:18:07.2547763Z Generating XML reports...
2023-01-11T22:18:07.2548216Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221411.xml
2023-01-11T22:18:07.2548586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2548743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2549121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2549309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2549328Z 
2023-01-11T22:18:07.2549434Z Running tests...
2023-01-11T22:18:07.2549692Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2550004Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2550307Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2551047Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.665s)
2023-01-11T22:18:07.2551068Z 
2023-01-11T22:18:07.2551324Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2551435Z Ran 1 test in 1.665s
2023-01-11T22:18:07.2551454Z 
2023-01-11T22:18:07.2551540Z OK (skipped=1)
2023-01-11T22:18:07.2551559Z 
2023-01-11T22:18:07.2551681Z Generating XML reports...
2023-01-11T22:18:07.2552375Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221415.xml
2023-01-11T22:18:07.2552769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2552945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2553322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2553509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2553529Z 
2023-01-11T22:18:07.2553637Z Running tests...
2023-01-11T22:18:07.2553880Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2554192Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2554503Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2554721Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47207
2023-01-11T22:18:07.2554942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47208
2023-01-11T22:18:07.2555311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2555482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2555857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2556047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2556394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2556565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2556932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2557211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2557519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2557771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2558170Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2558567Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2558794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2559007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2559154Z skip: Need at least 4 CUDA devices (4.285s)
2023-01-11T22:18:07.2559174Z 
2023-01-11T22:18:07.2559438Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2559555Z Ran 1 test in 4.285s
2023-01-11T22:18:07.2559574Z 
2023-01-11T22:18:07.2559682Z OK (skipped=1)
2023-01-11T22:18:07.2559701Z 
2023-01-11T22:18:07.2559823Z Generating XML reports...
2023-01-11T22:18:07.2560271Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221419.xml
2023-01-11T22:18:07.2560634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2560790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2561164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2561352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2561372Z 
2023-01-11T22:18:07.2561479Z Running tests...
2023-01-11T22:18:07.2561738Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2562057Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2562384Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2562603Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47312
2023-01-11T22:18:07.2562817Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47313
2023-01-11T22:18:07.2563169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2563343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2563718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2563910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2564278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2564446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2564817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2565006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2565235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2565477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2565872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2566266Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2566586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2566821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2566968Z skip: Need at least 4 CUDA devices (4.277s)
2023-01-11T22:18:07.2566988Z 
2023-01-11T22:18:07.2567253Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2567365Z Ran 1 test in 4.277s
2023-01-11T22:18:07.2567385Z 
2023-01-11T22:18:07.2567472Z OK (skipped=1)
2023-01-11T22:18:07.2567509Z 
2023-01-11T22:18:07.2567613Z Generating XML reports...
2023-01-11T22:18:07.2568062Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221426.xml
2023-01-11T22:18:07.2568429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2568610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2568984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2569171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2569190Z 
2023-01-11T22:18:07.2569297Z Running tests...
2023-01-11T22:18:07.2569557Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2569853Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2570137Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2570886Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.666s)
2023-01-11T22:18:07.2570910Z 
2023-01-11T22:18:07.2571168Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2571281Z Ran 1 test in 1.666s
2023-01-11T22:18:07.2571300Z 
2023-01-11T22:18:07.2571404Z OK (skipped=1)
2023-01-11T22:18:07.2571423Z 
2023-01-11T22:18:07.2571546Z Generating XML reports...
2023-01-11T22:18:07.2571986Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221433.xml
2023-01-11T22:18:07.2572353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2572528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2572888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2573078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2573103Z 
2023-01-11T22:18:07.2573212Z Running tests...
2023-01-11T22:18:07.2573477Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2573789Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2574057Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.2574077Z 
2023-01-11T22:18:07.2574332Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2574443Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2574462Z 
2023-01-11T22:18:07.2574549Z OK (skipped=1)
2023-01-11T22:18:07.2574584Z 
2023-01-11T22:18:07.2574688Z Generating XML reports...
2023-01-11T22:18:07.2575133Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221437.xml
2023-01-11T22:18:07.2575602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2575781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2576157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2576343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2576363Z 
2023-01-11T22:18:07.2576468Z Running tests...
2023-01-11T22:18:07.2576728Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2577020Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2577288Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.2577307Z 
2023-01-11T22:18:07.2577561Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2577673Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2577694Z 
2023-01-11T22:18:07.2577799Z OK (skipped=1)
2023-01-11T22:18:07.2577818Z 
2023-01-11T22:18:07.2577933Z Generating XML reports...
2023-01-11T22:18:07.2578373Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221440.xml
2023-01-11T22:18:07.2578739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2578913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2579270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2579457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2579476Z 
2023-01-11T22:18:07.2579582Z Running tests...
2023-01-11T22:18:07.2579839Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2580158Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2580431Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.2580450Z 
2023-01-11T22:18:07.2580705Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2580815Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2580833Z 
2023-01-11T22:18:07.2580934Z OK (skipped=1)
2023-01-11T22:18:07.2580953Z 
2023-01-11T22:18:07.2581056Z Generating XML reports...
2023-01-11T22:18:07.2581496Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221442.xml
2023-01-11T22:18:07.2581858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2582031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2582412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2582599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2582618Z 
2023-01-11T22:18:07.2582724Z Running tests...
2023-01-11T22:18:07.2582982Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2583275Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2583540Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.2583559Z 
2023-01-11T22:18:07.2583815Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2583924Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2583943Z 
2023-01-11T22:18:07.2584049Z OK (skipped=1)
2023-01-11T22:18:07.2584068Z 
2023-01-11T22:18:07.2584192Z Generating XML reports...
2023-01-11T22:18:07.2584729Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221444.xml
2023-01-11T22:18:07.2585092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2585260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2585612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2585799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2585819Z 
2023-01-11T22:18:07.2585922Z Running tests...
2023-01-11T22:18:07.2586176Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2586484Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2586742Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.2586767Z 
2023-01-11T22:18:07.2587023Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2587131Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2587151Z 
2023-01-11T22:18:07.2587255Z OK (skipped=1)
2023-01-11T22:18:07.2587273Z 
2023-01-11T22:18:07.2587377Z Generating XML reports...
2023-01-11T22:18:07.2587821Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221447.xml
2023-01-11T22:18:07.2588186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2588360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2588739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2588927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2588950Z 
2023-01-11T22:18:07.2589061Z Running tests...
2023-01-11T22:18:07.2589315Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2589610Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2589871Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.2589890Z 
2023-01-11T22:18:07.2590146Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2590256Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2590275Z 
2023-01-11T22:18:07.2590381Z OK (skipped=1)
2023-01-11T22:18:07.2590400Z 
2023-01-11T22:18:07.2590522Z Generating XML reports...
2023-01-11T22:18:07.2590964Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221449.xml
2023-01-11T22:18:07.2591335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2591512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2592099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2592302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2592322Z 
2023-01-11T22:18:07.2592431Z Running tests...
2023-01-11T22:18:07.2592704Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2593017Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2593284Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.2593303Z 
2023-01-11T22:18:07.2593558Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2593756Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2593775Z 
2023-01-11T22:18:07.2593935Z OK (skipped=1)
2023-01-11T22:18:07.2593956Z 
2023-01-11T22:18:07.2594066Z Generating XML reports...
2023-01-11T22:18:07.2594516Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221452.xml
2023-01-11T22:18:07.2594880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2595056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2595430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2595615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2595635Z 
2023-01-11T22:18:07.2595742Z Running tests...
2023-01-11T22:18:07.2596000Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2596300Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2596562Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.2596581Z 
2023-01-11T22:18:07.2596837Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2596946Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2596965Z 
2023-01-11T22:18:07.2597072Z OK (skipped=1)
2023-01-11T22:18:07.2597091Z 
2023-01-11T22:18:07.2597213Z Generating XML reports...
2023-01-11T22:18:07.2597657Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221454.xml
2023-01-11T22:18:07.2598026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2598196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2598558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2598753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2598772Z 
2023-01-11T22:18:07.2598879Z Running tests...
2023-01-11T22:18:07.2599137Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2599449Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2599702Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.2599722Z 
2023-01-11T22:18:07.2599977Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2600084Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2600103Z 
2023-01-11T22:18:07.2600206Z OK (skipped=1)
2023-01-11T22:18:07.2600225Z 
2023-01-11T22:18:07.2600327Z Generating XML reports...
2023-01-11T22:18:07.2600776Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221456.xml
2023-01-11T22:18:07.2601147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2601316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2601695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2601885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2601903Z 
2023-01-11T22:18:07.2602011Z Running tests...
2023-01-11T22:18:07.2602271Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2602580Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2602818Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.2602890Z 
2023-01-11T22:18:07.2603196Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2603311Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2603331Z 
2023-01-11T22:18:07.2603437Z OK (skipped=1)
2023-01-11T22:18:07.2603456Z 
2023-01-11T22:18:07.2603579Z Generating XML reports...
2023-01-11T22:18:07.2604025Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221459.xml
2023-01-11T22:18:07.2604390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2604565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2604924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2605114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2605133Z 
2023-01-11T22:18:07.2605244Z Running tests...
2023-01-11T22:18:07.2605506Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2605814Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2606076Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2606299Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47781
2023-01-11T22:18:07.2606516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47782
2023-01-11T22:18:07.2606881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2607035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2607415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2607605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2607963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2608132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2608503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2608692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2608940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2609168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2609570Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2609969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2610198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2610419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2610757Z STAGE:2023-01-11 22:15:06 47781:47781 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2611525Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1777: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2023-01-11T22:18:07.2611638Z   warnings.warn(
2023-01-11T22:18:07.2611969Z STAGE:2023-01-11 22:15:06 47782:47782 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2612783Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1777: UserWarning: torch.distributed.reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2023-01-11T22:18:07.2612921Z   warnings.warn(
2023-01-11T22:18:07.2613265Z STAGE:2023-01-11 22:15:06 47782:47782 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2613595Z STAGE:2023-01-11 22:15:06 47781:47781 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2613943Z STAGE:2023-01-11 22:15:06 47782:47782 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2614288Z STAGE:2023-01-11 22:15:06 47781:47781 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2614613Z STAGE:2023-01-11 22:15:06 47782:47782 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2614941Z STAGE:2023-01-11 22:15:06 47782:47782 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2615294Z STAGE:2023-01-11 22:15:06 47782:47782 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2615619Z STAGE:2023-01-11 22:15:06 47781:47781 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2615931Z STAGE:2023-01-11 22:15:06 47781:47781 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2616273Z STAGE:2023-01-11 22:15:06 47781:47781 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2616369Z ok (5.812s)
2023-01-11T22:18:07.2616390Z 
2023-01-11T22:18:07.2616649Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2616761Z Ran 1 test in 5.812s
2023-01-11T22:18:07.2616781Z 
2023-01-11T22:18:07.2616871Z OK
2023-01-11T22:18:07.2616890Z 
2023-01-11T22:18:07.2617012Z Generating XML reports...
2023-01-11T22:18:07.2617465Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221501.xml
2023-01-11T22:18:07.2617837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2617996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2618371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2618559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2618578Z 
2023-01-11T22:18:07.2618683Z Running tests...
2023-01-11T22:18:07.2618931Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2619238Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2619503Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.2619526Z 
2023-01-11T22:18:07.2619783Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2619876Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2619911Z 
2023-01-11T22:18:07.2619999Z OK (skipped=1)
2023-01-11T22:18:07.2620018Z 
2023-01-11T22:18:07.2620140Z Generating XML reports...
2023-01-11T22:18:07.2620587Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221509.xml
2023-01-11T22:18:07.2620952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2621124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2621497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2621686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2621706Z 
2023-01-11T22:18:07.2621865Z Running tests...
2023-01-11T22:18:07.2622165Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2622485Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2622759Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2622980Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47932
2023-01-11T22:18:07.2623194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47933
2023-01-11T22:18:07.2623561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2623734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2624109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2624304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2624647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2624817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2625184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2625370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2625612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2625856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2626253Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2626649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2626865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2627094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2627428Z STAGE:2023-01-11 22:15:17 47932:47932 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2627754Z STAGE:2023-01-11 22:15:17 47933:47933 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2628085Z STAGE:2023-01-11 22:15:18 47932:47932 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2628419Z STAGE:2023-01-11 22:15:18 47933:47933 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2628761Z STAGE:2023-01-11 22:15:18 47932:47932 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2629117Z STAGE:2023-01-11 22:15:18 47933:47933 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2629447Z STAGE:2023-01-11 22:15:18 47932:47932 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2629750Z STAGE:2023-01-11 22:15:18 47933:47933 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2630082Z STAGE:2023-01-11 22:15:18 47932:47932 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2630410Z STAGE:2023-01-11 22:15:18 47933:47933 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2630751Z STAGE:2023-01-11 22:15:18 47932:47932 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2631141Z STAGE:2023-01-11 22:15:18 47933:47933 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2631243Z ok (6.494s)
2023-01-11T22:18:07.2631264Z 
2023-01-11T22:18:07.2631525Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2631733Z Ran 1 test in 6.494s
2023-01-11T22:18:07.2631753Z 
2023-01-11T22:18:07.2632057Z OK
2023-01-11T22:18:07.2632080Z 
2023-01-11T22:18:07.2632204Z Generating XML reports...
2023-01-11T22:18:07.2632713Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221512.xml
2023-01-11T22:18:07.2633088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2633265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2633641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2633830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2633850Z 
2023-01-11T22:18:07.2633960Z Running tests...
2023-01-11T22:18:07.2634215Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2634520Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2634788Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2635008Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48051
2023-01-11T22:18:07.2635223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48052
2023-01-11T22:18:07.2635590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2635760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2636133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2636318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2636683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2636838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2637211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2637397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2637643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2637887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2638286Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2638687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2638922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2639146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2639229Z ok (5.667s)
2023-01-11T22:18:07.2639249Z 
2023-01-11T22:18:07.2639512Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2639624Z Ran 1 test in 5.667s
2023-01-11T22:18:07.2639644Z 
2023-01-11T22:18:07.2639732Z OK
2023-01-11T22:18:07.2639751Z 
2023-01-11T22:18:07.2639872Z Generating XML reports...
2023-01-11T22:18:07.2640323Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221521.xml
2023-01-11T22:18:07.2640693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2640864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2641366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2641563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2641582Z 
2023-01-11T22:18:07.2641690Z Running tests...
2023-01-11T22:18:07.2641952Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2642263Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2642516Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.2642535Z 
2023-01-11T22:18:07.2642791Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2642900Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2642919Z 
2023-01-11T22:18:07.2643024Z OK (skipped=1)
2023-01-11T22:18:07.2643043Z 
2023-01-11T22:18:07.2643146Z Generating XML reports...
2023-01-11T22:18:07.2643596Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221529.xml
2023-01-11T22:18:07.2643956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2644130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2644504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2644692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2644711Z 
2023-01-11T22:18:07.2644818Z Running tests...
2023-01-11T22:18:07.2645074Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2645369Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2645628Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2645850Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48199
2023-01-11T22:18:07.2646065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48200
2023-01-11T22:18:07.2646431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2646602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2646976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2647164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2647525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2647680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2648052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2648243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2648488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2648731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2649128Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2649523Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2649752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2649982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2650440Z STAGE:2023-01-11 22:15:36 48199:48199 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2650769Z STAGE:2023-01-11 22:15:36 48200:48200 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2651102Z STAGE:2023-01-11 22:15:36 48200:48200 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2651421Z STAGE:2023-01-11 22:15:36 48199:48199 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2651765Z STAGE:2023-01-11 22:15:36 48200:48200 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2652109Z STAGE:2023-01-11 22:15:36 48199:48199 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2652435Z STAGE:2023-01-11 22:15:37 48199:48199 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2652765Z STAGE:2023-01-11 22:15:37 48199:48199 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2653113Z STAGE:2023-01-11 22:15:37 48199:48199 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2653422Z STAGE:2023-01-11 22:15:37 48200:48200 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2653751Z STAGE:2023-01-11 22:15:37 48200:48200 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2654097Z STAGE:2023-01-11 22:15:37 48200:48200 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2654198Z ok (5.766s)
2023-01-11T22:18:07.2654218Z 
2023-01-11T22:18:07.2654481Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2654591Z Ran 1 test in 5.767s
2023-01-11T22:18:07.2654611Z 
2023-01-11T22:18:07.2654701Z OK
2023-01-11T22:18:07.2654721Z 
2023-01-11T22:18:07.2654843Z Generating XML reports...
2023-01-11T22:18:07.2655278Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221531.xml
2023-01-11T22:18:07.2655654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2655829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2656206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2656393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2656413Z 
2023-01-11T22:18:07.2656518Z Running tests...
2023-01-11T22:18:07.2656777Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2657091Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2657357Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2657557Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48317
2023-01-11T22:18:07.2657779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48318
2023-01-11T22:18:07.2658147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2658316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2658690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2658878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2659241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2659409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2659763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2660042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2660293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2660540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2660938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2661335Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2661565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2661790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2662122Z STAGE:2023-01-11 22:15:45 48318:48318 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2662434Z STAGE:2023-01-11 22:15:45 48317:48317 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2662769Z STAGE:2023-01-11 22:15:45 48318:48318 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2663097Z STAGE:2023-01-11 22:15:45 48317:48317 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2663441Z STAGE:2023-01-11 22:15:45 48318:48318 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2663783Z STAGE:2023-01-11 22:15:45 48317:48317 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2664106Z STAGE:2023-01-11 22:15:45 48318:48318 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2664432Z STAGE:2023-01-11 22:15:45 48318:48318 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2664779Z STAGE:2023-01-11 22:15:45 48318:48318 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2665106Z STAGE:2023-01-11 22:15:45 48317:48317 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2665419Z STAGE:2023-01-11 22:15:45 48317:48317 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2665761Z STAGE:2023-01-11 22:15:45 48317:48317 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2665860Z ok (5.756s)
2023-01-11T22:18:07.2665880Z 
2023-01-11T22:18:07.2666145Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2666255Z Ran 1 test in 5.756s
2023-01-11T22:18:07.2666275Z 
2023-01-11T22:18:07.2666366Z OK
2023-01-11T22:18:07.2666384Z 
2023-01-11T22:18:07.2666506Z Generating XML reports...
2023-01-11T22:18:07.2666955Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221540.xml
2023-01-11T22:18:07.2667329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2667489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2667865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2668056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2668075Z 
2023-01-11T22:18:07.2668179Z Running tests...
2023-01-11T22:18:07.2668435Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2668750Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2669012Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.2669032Z 
2023-01-11T22:18:07.2669283Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2669431Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2669468Z 
2023-01-11T22:18:07.2669595Z OK (skipped=1)
2023-01-11T22:18:07.2669616Z 
2023-01-11T22:18:07.2669744Z Generating XML reports...
2023-01-11T22:18:07.2670194Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221548.xml
2023-01-11T22:18:07.2670563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2670736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2671112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2671300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2671319Z 
2023-01-11T22:18:07.2671427Z Running tests...
2023-01-11T22:18:07.2671668Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2672219Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2672485Z   test_scatter (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.2672506Z 
2023-01-11T22:18:07.2672774Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2672881Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2672901Z 
2023-01-11T22:18:07.2673004Z OK (skipped=1)
2023-01-11T22:18:07.2673024Z 
2023-01-11T22:18:07.2673145Z Generating XML reports...
2023-01-11T22:18:07.2673591Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221550.xml
2023-01-11T22:18:07.2673957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2674113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2674488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2674676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2674695Z 
2023-01-11T22:18:07.2674799Z Running tests...
2023-01-11T22:18:07.2675052Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2675363Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2675624Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.2675643Z 
2023-01-11T22:18:07.2675898Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2675989Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2676027Z 
2023-01-11T22:18:07.2676114Z OK (skipped=1)
2023-01-11T22:18:07.2676133Z 
2023-01-11T22:18:07.2676254Z Generating XML reports...
2023-01-11T22:18:07.2676702Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221553.xml
2023-01-11T22:18:07.2677075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2677247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2677624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2677811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2677831Z 
2023-01-11T22:18:07.2677935Z Running tests...
2023-01-11T22:18:07.2678177Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2678488Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2678751Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.2678849Z 
2023-01-11T22:18:07.2679173Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2679293Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2679313Z 
2023-01-11T22:18:07.2679421Z OK (skipped=1)
2023-01-11T22:18:07.2679440Z 
2023-01-11T22:18:07.2679560Z Generating XML reports...
2023-01-11T22:18:07.2680014Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221555.xml
2023-01-11T22:18:07.2680377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2680533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2680913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2681102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2681122Z 
2023-01-11T22:18:07.2681233Z Running tests...
2023-01-11T22:18:07.2681494Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2681804Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2682053Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2682275Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48567
2023-01-11T22:18:07.2682476Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48568
2023-01-11T22:18:07.2682844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2683017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2683389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2683582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2683952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2684123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2684496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2684684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2684915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2685157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2685555Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2685952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2686185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2686412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2686749Z STAGE:2023-01-11 22:16:02 48568:48568 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2687075Z STAGE:2023-01-11 22:16:02 48567:48567 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2687409Z STAGE:2023-01-11 22:16:03 48567:48567 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2687739Z STAGE:2023-01-11 22:16:03 48567:48567 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2688067Z STAGE:2023-01-11 22:16:03 48568:48568 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2688514Z STAGE:2023-01-11 22:16:03 48568:48568 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2688853Z STAGE:2023-01-11 22:16:03 48567:48567 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2689180Z STAGE:2023-01-11 22:16:03 48568:48568 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2689506Z STAGE:2023-01-11 22:16:03 48568:48568 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2689846Z STAGE:2023-01-11 22:16:03 48568:48568 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2690177Z STAGE:2023-01-11 22:16:03 48567:48567 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2690518Z STAGE:2023-01-11 22:16:03 48567:48567 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2690601Z ok (6.459s)
2023-01-11T22:18:07.2690622Z 
2023-01-11T22:18:07.2690883Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2690999Z Ran 1 test in 6.459s
2023-01-11T22:18:07.2691018Z 
2023-01-11T22:18:07.2691107Z OK
2023-01-11T22:18:07.2691126Z 
2023-01-11T22:18:07.2691246Z Generating XML reports...
2023-01-11T22:18:07.2691693Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221557.xml
2023-01-11T22:18:07.2692060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2692235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2692595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2692782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2692801Z 
2023-01-11T22:18:07.2692907Z Running tests...
2023-01-11T22:18:07.2693167Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2693487Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2693756Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2693973Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48690
2023-01-11T22:18:07.2694188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48691
2023-01-11T22:18:07.2694557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2694712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2695084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2695272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2695641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2695813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2696188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2696370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2696617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2696844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2697244Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2697633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2697956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2698188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2698525Z STAGE:2023-01-11 22:16:11 48690:48690 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2698851Z STAGE:2023-01-11 22:16:11 48691:48691 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2699184Z STAGE:2023-01-11 22:16:12 48690:48690 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2699533Z STAGE:2023-01-11 22:16:12 48690:48690 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2699847Z STAGE:2023-01-11 22:16:12 48691:48691 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2700189Z STAGE:2023-01-11 22:16:12 48691:48691 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2700522Z STAGE:2023-01-11 22:16:12 48690:48690 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2700843Z STAGE:2023-01-11 22:16:12 48691:48691 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2701178Z STAGE:2023-01-11 22:16:12 48691:48691 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2701520Z STAGE:2023-01-11 22:16:12 48691:48691 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2701848Z STAGE:2023-01-11 22:16:12 48690:48690 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2702190Z STAGE:2023-01-11 22:16:12 48690:48690 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2702290Z ok (6.551s)
2023-01-11T22:18:07.2702310Z 
2023-01-11T22:18:07.2702553Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2702661Z Ran 1 test in 6.551s
2023-01-11T22:18:07.2702683Z 
2023-01-11T22:18:07.2702771Z OK
2023-01-11T22:18:07.2702793Z 
2023-01-11T22:18:07.2702917Z Generating XML reports...
2023-01-11T22:18:07.2703367Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221606.xml
2023-01-11T22:18:07.2703736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2703908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2704281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2704455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2704491Z 
2023-01-11T22:18:07.2704581Z Running tests...
2023-01-11T22:18:07.2704842Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2705152Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2705423Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.2705443Z 
2023-01-11T22:18:07.2705699Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2705808Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2705827Z 
2023-01-11T22:18:07.2705933Z OK (skipped=1)
2023-01-11T22:18:07.2705952Z 
2023-01-11T22:18:07.2706072Z Generating XML reports...
2023-01-11T22:18:07.2706505Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221615.xml
2023-01-11T22:18:07.2706873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2707048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2707424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2707702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2707722Z 
2023-01-11T22:18:07.2707832Z Running tests...
2023-01-11T22:18:07.2708095Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2708407Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2708665Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... skip: Nccl does not support CPU tensors (0.002s)
2023-01-11T22:18:07.2708685Z 
2023-01-11T22:18:07.2708927Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2709037Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2709056Z 
2023-01-11T22:18:07.2709162Z OK (skipped=1)
2023-01-11T22:18:07.2709181Z 
2023-01-11T22:18:07.2709301Z Generating XML reports...
2023-01-11T22:18:07.2709747Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221618.xml
2023-01-11T22:18:07.2710120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2710291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2710662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2710846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2710866Z 
2023-01-11T22:18:07.2710954Z Running tests...
2023-01-11T22:18:07.2711211Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2711524Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2712213Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'gloo'} (0.002s)
2023-01-11T22:18:07.2712243Z 
2023-01-11T22:18:07.2712527Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2712639Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2712659Z 
2023-01-11T22:18:07.2712763Z OK (skipped=1)
2023-01-11T22:18:07.2712783Z 
2023-01-11T22:18:07.2712903Z Generating XML reports...
2023-01-11T22:18:07.2713332Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221620.xml
2023-01-11T22:18:07.2713696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2713870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2714247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2714436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2714455Z 
2023-01-11T22:18:07.2714565Z Running tests...
2023-01-11T22:18:07.2714826Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2715136Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2715396Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... skip: Nccl send/recv tested by test_send_recv_nccl (0.002s)
2023-01-11T22:18:07.2715416Z 
2023-01-11T22:18:07.2715657Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2715765Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2715784Z 
2023-01-11T22:18:07.2715887Z OK (skipped=1)
2023-01-11T22:18:07.2715905Z 
2023-01-11T22:18:07.2716027Z Generating XML reports...
2023-01-11T22:18:07.2716461Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221623.xml
2023-01-11T22:18:07.2716824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2717135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2717520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2717708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2717729Z 
2023-01-11T22:18:07.2717818Z Running tests...
2023-01-11T22:18:07.2718081Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2718393Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2718677Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2023-01-11T22:18:07.2718696Z 
2023-01-11T22:18:07.2718945Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2719053Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2719072Z 
2023-01-11T22:18:07.2719184Z OK (skipped=1)
2023-01-11T22:18:07.2719203Z 
2023-01-11T22:18:07.2719326Z Generating XML reports...
2023-01-11T22:18:07.2719772Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221625.xml
2023-01-11T22:18:07.2720123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2720297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2720671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2720858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2720878Z 
2023-01-11T22:18:07.2720984Z Running tests...
2023-01-11T22:18:07.2721242Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2721560Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2721877Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2023-01-11T22:18:07.2721898Z 
2023-01-11T22:18:07.2722152Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2722244Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2722263Z 
2023-01-11T22:18:07.2722367Z OK (skipped=1)
2023-01-11T22:18:07.2722386Z 
2023-01-11T22:18:07.2722508Z Generating XML reports...
2023-01-11T22:18:07.2722949Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221627.xml
2023-01-11T22:18:07.2723314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2723486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2723864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2724058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2724077Z 
2023-01-11T22:18:07.2724166Z Running tests...
2023-01-11T22:18:07.2724424Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2724734Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2725033Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support send/recv from any source (0.002s)
2023-01-11T22:18:07.2725052Z 
2023-01-11T22:18:07.2725305Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2725413Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2725432Z 
2023-01-11T22:18:07.2725537Z OK (skipped=1)
2023-01-11T22:18:07.2725556Z 
2023-01-11T22:18:07.2725676Z Generating XML reports...
2023-01-11T22:18:07.2726168Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221630.xml
2023-01-11T22:18:07.2726569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2726743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2727118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2727305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2727325Z 
2023-01-11T22:18:07.2727431Z Running tests...
2023-01-11T22:18:07.2727690Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2728005Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2728294Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2023-01-11T22:18:07.2728316Z 
2023-01-11T22:18:07.2728573Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2728666Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2728685Z 
2023-01-11T22:18:07.2728789Z OK (skipped=1)
2023-01-11T22:18:07.2728808Z 
2023-01-11T22:18:07.2728927Z Generating XML reports...
2023-01-11T22:18:07.2729372Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221632.xml
2023-01-11T22:18:07.2729740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2729911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2730288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2730477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2730500Z 
2023-01-11T22:18:07.2730609Z Running tests...
2023-01-11T22:18:07.2730898Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2731212Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2731465Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2731685Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49077
2023-01-11T22:18:07.2731901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49078
2023-01-11T22:18:07.2732269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2732441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2732817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2732997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2733357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2733529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2733903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2734087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2734330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2734575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2734974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2735475Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2735696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2735924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2736024Z ok (5.200s)
2023-01-11T22:18:07.2736044Z 
2023-01-11T22:18:07.2736303Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2736413Z Ran 1 test in 5.201s
2023-01-11T22:18:07.2736433Z 
2023-01-11T22:18:07.2736524Z OK
2023-01-11T22:18:07.2736543Z 
2023-01-11T22:18:07.2736664Z Generating XML reports...
2023-01-11T22:18:07.2737115Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221635.xml
2023-01-11T22:18:07.2737467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2737645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2738019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2738211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2738230Z 
2023-01-11T22:18:07.2738335Z Running tests...
2023-01-11T22:18:07.2738594Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2738909Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2739191Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2739410Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49195
2023-01-11T22:18:07.2739610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49196
2023-01-11T22:18:07.2739986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2740159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2740529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2740716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2741076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2741246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2741619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2741790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2742038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2742286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2742680Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2743075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2743304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2743530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2743865Z STAGE:2023-01-11 22:16:47 49196:49196 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2744183Z STAGE:2023-01-11 22:16:47 49195:49195 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2744597Z STAGE:2023-01-11 22:16:47 49196:49196 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2744932Z STAGE:2023-01-11 22:16:47 49195:49195 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2745280Z STAGE:2023-01-11 22:16:47 49196:49196 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2745627Z STAGE:2023-01-11 22:16:47 49195:49195 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2745727Z ok (5.770s)
2023-01-11T22:18:07.2745746Z 
2023-01-11T22:18:07.2746008Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2746120Z Ran 1 test in 5.770s
2023-01-11T22:18:07.2746139Z 
2023-01-11T22:18:07.2746227Z OK
2023-01-11T22:18:07.2746246Z 
2023-01-11T22:18:07.2746367Z Generating XML reports...
2023-01-11T22:18:07.2746802Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221642.xml
2023-01-11T22:18:07.2747177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2747349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2747724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2747915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2747934Z 
2023-01-11T22:18:07.2748039Z Running tests...
2023-01-11T22:18:07.2748295Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2748605Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2748865Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2749083Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49317
2023-01-11T22:18:07.2749308Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49318
2023-01-11T22:18:07.2749677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2749848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2750225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2750414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2750777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2750950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2751303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2751492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2751736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2752211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2752629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2753023Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2753252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2753481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2753817Z STAGE:2023-01-11 22:16:56 49318:49318 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2754286Z STAGE:2023-01-11 22:16:56 49317:49317 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:18:07.2754631Z STAGE:2023-01-11 22:16:56 49318:49318 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2754962Z STAGE:2023-01-11 22:16:56 49317:49317 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:18:07.2755309Z STAGE:2023-01-11 22:16:56 49318:49318 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2755652Z STAGE:2023-01-11 22:16:56 49317:49317 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:18:07.2755753Z ok (5.796s)
2023-01-11T22:18:07.2755775Z 
2023-01-11T22:18:07.2756033Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2756144Z Ran 1 test in 5.797s
2023-01-11T22:18:07.2756163Z 
2023-01-11T22:18:07.2756235Z OK
2023-01-11T22:18:07.2756270Z 
2023-01-11T22:18:07.2756375Z Generating XML reports...
2023-01-11T22:18:07.2756831Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221651.xml
2023-01-11T22:18:07.2757194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2757368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2757745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2757935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2757954Z 
2023-01-11T22:18:07.2758059Z Running tests...
2023-01-11T22:18:07.2758318Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2758614Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2758901Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2023-01-11T22:18:07.2758923Z 
2023-01-11T22:18:07.2759183Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2759296Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2759315Z 
2023-01-11T22:18:07.2759423Z OK (skipped=1)
2023-01-11T22:18:07.2759441Z 
2023-01-11T22:18:07.2759563Z Generating XML reports...
2023-01-11T22:18:07.2760011Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221659.xml
2023-01-11T22:18:07.2760379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2760550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2760908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2761095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2761120Z 
2023-01-11T22:18:07.2761221Z Running tests...
2023-01-11T22:18:07.2761479Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2761788Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2762060Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2023-01-11T22:18:07.2762080Z 
2023-01-11T22:18:07.2762337Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2762446Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2762465Z 
2023-01-11T22:18:07.2762552Z OK (skipped=1)
2023-01-11T22:18:07.2762588Z 
2023-01-11T22:18:07.2762692Z Generating XML reports...
2023-01-11T22:18:07.2763138Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221701.xml
2023-01-11T22:18:07.2763603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2763781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2764161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2764349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2764369Z 
2023-01-11T22:18:07.2764476Z Running tests...
2023-01-11T22:18:07.2764733Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2765027Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2765325Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2023-01-11T22:18:07.2765344Z 
2023-01-11T22:18:07.2765600Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2765716Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2765735Z 
2023-01-11T22:18:07.2765837Z OK (skipped=1)
2023-01-11T22:18:07.2765856Z 
2023-01-11T22:18:07.2765974Z Generating XML reports...
2023-01-11T22:18:07.2766416Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221704.xml
2023-01-11T22:18:07.2766784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2766956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2767314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2767502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2767522Z 
2023-01-11T22:18:07.2767628Z Running tests...
2023-01-11T22:18:07.2767885Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2768207Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2768500Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL send/recv tested by test_send_recv_nccl (0.002s)
2023-01-11T22:18:07.2768519Z 
2023-01-11T22:18:07.2768775Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2768887Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2768906Z 
2023-01-11T22:18:07.2769012Z OK (skipped=1)
2023-01-11T22:18:07.2769030Z 
2023-01-11T22:18:07.2769136Z Generating XML reports...
2023-01-11T22:18:07.2769582Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221706.xml
2023-01-11T22:18:07.2769950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2770122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2770503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2770694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2770713Z 
2023-01-11T22:18:07.2770814Z Running tests...
2023-01-11T22:18:07.2771076Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2771372Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2771657Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2023-01-11T22:18:07.2771676Z 
2023-01-11T22:18:07.2771932Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2772041Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2772060Z 
2023-01-11T22:18:07.2772166Z OK (skipped=1)
2023-01-11T22:18:07.2772235Z 
2023-01-11T22:18:07.2772361Z Generating XML reports...
2023-01-11T22:18:07.2772847Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221709.xml
2023-01-11T22:18:07.2773217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2773392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2773752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2773939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2773959Z 
2023-01-11T22:18:07.2774065Z Running tests...
2023-01-11T22:18:07.2774325Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2774637Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2774938Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Gloo backend support sparse all reduce (0.002s)
2023-01-11T22:18:07.2774957Z 
2023-01-11T22:18:07.2775216Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2775328Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2775347Z 
2023-01-11T22:18:07.2775453Z OK (skipped=1)
2023-01-11T22:18:07.2775471Z 
2023-01-11T22:18:07.2775575Z Generating XML reports...
2023-01-11T22:18:07.2776014Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221711.xml
2023-01-11T22:18:07.2776380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2776552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2776926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2777118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2777137Z 
2023-01-11T22:18:07.2777249Z Running tests...
2023-01-11T22:18:07.2777505Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2777815Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2778068Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2778285Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49637
2023-01-11T22:18:07.2778499Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49638
2023-01-11T22:18:07.2778860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2779034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2779407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2779604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2779969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2780124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2780498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2780687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2780936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2781178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2781624Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2782075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2782304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2782531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2782614Z ok (6.111s)
2023-01-11T22:18:07.2782634Z 
2023-01-11T22:18:07.2782894Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2783004Z Ran 1 test in 6.111s
2023-01-11T22:18:07.2783023Z 
2023-01-11T22:18:07.2783111Z OK
2023-01-11T22:18:07.2783131Z 
2023-01-11T22:18:07.2783254Z Generating XML reports...
2023-01-11T22:18:07.2783703Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221713.xml
2023-01-11T22:18:07.2784083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2784260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2784621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2784810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2784829Z 
2023-01-11T22:18:07.2784935Z Running tests...
2023-01-11T22:18:07.2785194Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2785507Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2785781Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... skip: nccl does not support DDP on CPU models (0.002s)
2023-01-11T22:18:07.2785800Z 
2023-01-11T22:18:07.2786058Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2786173Z Ran 1 test in 0.002s
2023-01-11T22:18:07.2786195Z 
2023-01-11T22:18:07.2786303Z OK (skipped=1)
2023-01-11T22:18:07.2786322Z 
2023-01-11T22:18:07.2786426Z Generating XML reports...
2023-01-11T22:18:07.2786871Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221722.xml
2023-01-11T22:18:07.2787239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2787411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2787786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2787974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2787993Z 
2023-01-11T22:18:07.2788099Z Running tests...
2023-01-11T22:18:07.2788355Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2788674Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2788916Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2789135Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49789
2023-01-11T22:18:07.2789355Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49790
2023-01-11T22:18:07.2789719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2789892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2790267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2790456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2790863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2791097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2791471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2791653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2792124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2792380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2792790Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2793186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2793422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2793648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2793730Z ok (5.713s)
2023-01-11T22:18:07.2793751Z 
2023-01-11T22:18:07.2794012Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2794117Z Ran 1 test in 5.714s
2023-01-11T22:18:07.2794136Z 
2023-01-11T22:18:07.2794226Z OK
2023-01-11T22:18:07.2794245Z 
2023-01-11T22:18:07.2794365Z Generating XML reports...
2023-01-11T22:18:07.2794810Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221724.xml
2023-01-11T22:18:07.2795178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2795348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2795734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2795905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2795923Z 
2023-01-11T22:18:07.2796030Z Running tests...
2023-01-11T22:18:07.2796288Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2796596Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2796891Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2797109Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49904
2023-01-11T22:18:07.2797326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49905
2023-01-11T22:18:07.2797689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2797851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2798228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2798420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2798778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2798947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2799318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2799504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2799747Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2799991Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2800527Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2800934Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2801162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2801388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2802174Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:18:07.2802955Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:18:07.2803060Z ok (6.085s)
2023-01-11T22:18:07.2803080Z 
2023-01-11T22:18:07.2803347Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2803458Z Ran 1 test in 6.085s
2023-01-11T22:18:07.2803477Z 
2023-01-11T22:18:07.2803568Z OK
2023-01-11T22:18:07.2803590Z 
2023-01-11T22:18:07.2803717Z Generating XML reports...
2023-01-11T22:18:07.2804171Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221733.xml
2023-01-11T22:18:07.2804521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2804695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2805071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2805260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2805279Z 
2023-01-11T22:18:07.2805384Z Running tests...
2023-01-11T22:18:07.2805643Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2805953Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2806242Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2806447Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50023
2023-01-11T22:18:07.2806669Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50024
2023-01-11T22:18:07.2807039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2807212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2807588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2807777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2808141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2808313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2808781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2808959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2809207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2809450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2809852Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2810244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2810474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2810699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2810945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:18:07.2811186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:18:07.2811563Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.2811955Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.2812194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:18:07.2812428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:18:07.2812822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:18:07.2813219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:18:07.2813321Z ok (11.061s)
2023-01-11T22:18:07.2813341Z 
2023-01-11T22:18:07.2813600Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2813710Z Ran 1 test in 11.062s
2023-01-11T22:18:07.2813730Z 
2023-01-11T22:18:07.2813802Z OK
2023-01-11T22:18:07.2813821Z 
2023-01-11T22:18:07.2813946Z Generating XML reports...
2023-01-11T22:18:07.2814395Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221741.xml
2023-01-11T22:18:07.2814761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2814932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2815304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2815499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2815519Z 
2023-01-11T22:18:07.2815628Z Running tests...
2023-01-11T22:18:07.2815868Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2816181Z Test results will be stored in test-reports/dist-nccl/distributed.test_distributed_spawn
2023-01-11T22:18:07.2816469Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:18:07.2816685Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50145
2023-01-11T22:18:07.2816903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50146
2023-01-11T22:18:07.2817267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2817435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2817917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2818113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2818460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:18:07.2818628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:18:07.2818999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:18:07.2819179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:18:07.2819421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:18:07.2819660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:18:07.2820053Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2820436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:18:07.2820653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:18:07.2820863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:18:07.2821092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:18:07.2821332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:18:07.2821726Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.2822120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:18:07.2822363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:18:07.2822597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:18:07.2822992Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:18:07.2823381Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:18:07.2823464Z ok (10.901s)
2023-01-11T22:18:07.2823483Z 
2023-01-11T22:18:07.2823747Z ----------------------------------------------------------------------
2023-01-11T22:18:07.2823859Z Ran 1 test in 10.901s
2023-01-11T22:18:07.2823879Z 
2023-01-11T22:18:07.2823969Z OK
2023-01-11T22:18:07.2823988Z 
2023-01-11T22:18:07.2824110Z Generating XML reports...
2023-01-11T22:18:07.2824568Z Generated XML report: test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221755.xml
2023-01-11T22:18:07.2824587Z 
2023-01-11T22:18:07.2825067Z ##[endgroup]
2023-01-11T22:18:07.2825539Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_qe58onb1)
2023-01-11T22:18:07.2825560Z 
2023-01-11T22:18:07.2825668Z Shard 2: gloo should be run in 1
2023-01-11T22:18:07.2825789Z Shard 2: ucc should be run in 3
2023-01-11T22:18:07.2826066Z Running distributed/pipeline/sync/test_worker ... [2023-01-11 22:18:07.044509]
2023-01-11T22:18:07.2826458Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_worker.py', '-v'] ... [2023-01-11 22:18:07.044736]
2023-01-11T22:18:09.9436563Z 
2023-01-11T22:18:09.9437096Z Expand the folded group to see the log file of distributed/pipeline/sync/test_worker
2023-01-11T22:18:09.9438992Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_worker (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_worker_cat6y34c)
2023-01-11T22:18:09.9439558Z ============================= test session starts ==============================
2023-01-11T22:18:09.9440369Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T22:18:09.9440916Z cachedir: .pytest_cache
2023-01-11T22:18:09.9441707Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T22:18:09.9442147Z torch: 2.0.0a0+git8419ddd
2023-01-11T22:18:09.9442652Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T22:18:09.9443286Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T22:18:09.9443744Z collecting ... collected 6 items
2023-01-11T22:18:09.9444788Z Running 6 items in this shard: test/distributed/pipeline/sync/test_worker.py::test_compute_multithreading, test/distributed/pipeline/sync/test_worker.py::test_compute_success, test/distributed/pipeline/sync/test_worker.py::test_compute_exception, test/distributed/pipeline/sync/test_worker.py::test_grad_mode[True], test/distributed/pipeline/sync/test_worker.py::test_grad_mode[False], test/distributed/pipeline/sync/test_worker.py::test_worker_per_device
2023-01-11T22:18:09.9445750Z 
2023-01-11T22:18:09.9445988Z distributed/pipeline/sync/test_worker.py::test_compute_multithreading PASSED [ 16%]
2023-01-11T22:18:09.9446653Z distributed/pipeline/sync/test_worker.py::test_compute_success PASSED    [ 33%]
2023-01-11T22:18:09.9447087Z distributed/pipeline/sync/test_worker.py::test_compute_exception PASSED  [ 50%]
2023-01-11T22:18:09.9447518Z distributed/pipeline/sync/test_worker.py::test_grad_mode[True] PASSED    [ 66%]
2023-01-11T22:18:09.9448171Z distributed/pipeline/sync/test_worker.py::test_grad_mode[False] PASSED   [ 83%]
2023-01-11T22:18:09.9448594Z distributed/pipeline/sync/test_worker.py::test_worker_per_device PASSED  [100%]
2023-01-11T22:18:09.9448881Z 
2023-01-11T22:18:09.9449171Z ============================== 6 passed in 0.07s ===============================
2023-01-11T22:18:09.9449401Z 
2023-01-11T22:18:09.9449727Z ##[endgroup]
2023-01-11T22:18:09.9450609Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_worker (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_worker_cat6y34c)
2023-01-11T22:18:09.9450979Z 
2023-01-11T22:18:09.9451275Z Running distributed/pipeline/sync/test_pipeline ... [2023-01-11 22:18:09.943709]
2023-01-11T22:18:09.9452125Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_pipeline.py', '-v'] ... [2023-01-11 22:18:09.943985]
2023-01-11T22:18:12.3401583Z 
2023-01-11T22:18:12.3402486Z Expand the folded group to see the log file of distributed/pipeline/sync/test_pipeline
2023-01-11T22:18:12.3403544Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_pipeline (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_pipeline_90il3872)
2023-01-11T22:18:12.3404233Z ============================= test session starts ==============================
2023-01-11T22:18:12.3404845Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T22:18:12.3405301Z cachedir: .pytest_cache
2023-01-11T22:18:12.3406144Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T22:18:12.3406579Z torch: 2.0.0a0+git8419ddd
2023-01-11T22:18:12.3406904Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T22:18:12.3407476Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T22:18:12.3407854Z collecting ... collected 1 item
2023-01-11T22:18:12.3408254Z Running 1 items in this shard: test/distributed/pipeline/sync/test_pipeline.py::test_clock_cycles
2023-01-11T22:18:12.3408807Z 
2023-01-11T22:18:12.3409107Z distributed/pipeline/sync/test_pipeline.py::test_clock_cycles PASSED     [100%]
2023-01-11T22:18:12.3409366Z 
2023-01-11T22:18:12.3409524Z ============================== 1 passed in 0.03s ===============================
2023-01-11T22:18:12.3409712Z 
2023-01-11T22:18:12.3410249Z ##[endgroup]
2023-01-11T22:18:12.3410910Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_pipeline (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_pipeline_90il3872)
2023-01-11T22:18:12.3411298Z 
2023-01-11T22:18:12.3411596Z Running distributed/pipeline/sync/test_microbatch ... [2023-01-11 22:18:12.340243]
2023-01-11T22:18:12.3412192Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_microbatch.py', '-v'] ... [2023-01-11 22:18:12.340497]
2023-01-11T22:18:14.7698084Z 
2023-01-11T22:18:14.7698684Z Expand the folded group to see the log file of distributed/pipeline/sync/test_microbatch
2023-01-11T22:18:14.7700260Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_microbatch (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_microbatch_lnqt45ey)
2023-01-11T22:18:14.7700812Z ============================= test session starts ==============================
2023-01-11T22:18:14.7701581Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T22:18:14.7701920Z cachedir: .pytest_cache
2023-01-11T22:18:14.7702719Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T22:18:14.7703170Z torch: 2.0.0a0+git8419ddd
2023-01-11T22:18:14.7703629Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T22:18:14.7704231Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T22:18:14.7704757Z collecting ... collected 10 items
2023-01-11T22:18:14.7706129Z Running 10 items in this shard: test/distributed/pipeline/sync/test_microbatch.py::test_batch_atomic, test/distributed/pipeline/sync/test_microbatch.py::test_batch_non_atomic, test/distributed/pipeline/sync/test_microbatch.py::test_batch_call, test/distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_index, test/distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_slice, test/distributed/pipeline/sync/test_microbatch.py::test_check, test/distributed/pipeline/sync/test_microbatch.py::test_gather_tensors, test/distributed/pipeline/sync/test_microbatch.py::test_gather_tuples, test/distributed/pipeline/sync/test_microbatch.py::test_scatter_tensor, test/distributed/pipeline/sync/test_microbatch.py::test_scatter_multiple_tensors
2023-01-11T22:18:14.7707224Z 
2023-01-11T22:18:14.7707443Z distributed/pipeline/sync/test_microbatch.py::test_batch_atomic PASSED   [ 10%]
2023-01-11T22:18:14.7707890Z distributed/pipeline/sync/test_microbatch.py::test_batch_non_atomic PASSED [ 20%]
2023-01-11T22:18:14.7708334Z distributed/pipeline/sync/test_microbatch.py::test_batch_call PASSED     [ 30%]
2023-01-11T22:18:14.7708772Z distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_index PASSED [ 40%]
2023-01-11T22:18:14.7709235Z distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_slice PASSED [ 50%]
2023-01-11T22:18:14.7709675Z distributed/pipeline/sync/test_microbatch.py::test_check PASSED          [ 60%]
2023-01-11T22:18:14.7710083Z distributed/pipeline/sync/test_microbatch.py::test_gather_tensors PASSED [ 70%]
2023-01-11T22:18:14.7710530Z distributed/pipeline/sync/test_microbatch.py::test_gather_tuples PASSED  [ 80%]
2023-01-11T22:18:14.7710966Z distributed/pipeline/sync/test_microbatch.py::test_scatter_tensor PASSED [ 90%]
2023-01-11T22:18:14.7711420Z distributed/pipeline/sync/test_microbatch.py::test_scatter_multiple_tensors PASSED [100%]
2023-01-11T22:18:14.7711686Z 
2023-01-11T22:18:14.7711824Z ============================== 10 passed in 0.08s ==============================
2023-01-11T22:18:14.7712723Z 
2023-01-11T22:18:14.7713311Z ##[endgroup]
2023-01-11T22:18:14.7714076Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_microbatch (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_microbatch_lnqt45ey)
2023-01-11T22:18:14.7714480Z 
2023-01-11T22:18:14.7714774Z Running distributed/pipeline/sync/test_deferred_batch_norm ... [2023-01-11 22:18:14.769916]
2023-01-11T22:18:14.7715416Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_deferred_batch_norm.py', '-v'] ... [2023-01-11 22:18:14.770170]
2023-01-11T22:18:17.7048563Z 
2023-01-11T22:18:17.7049421Z Expand the folded group to see the log file of distributed/pipeline/sync/test_deferred_batch_norm
2023-01-11T22:18:17.7051217Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_deferred_batch_norm (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_deferred_batch_norm_wjmd9mmf)
2023-01-11T22:18:17.7051953Z ============================= test session starts ==============================
2023-01-11T22:18:17.7052624Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T22:18:17.7052986Z cachedir: .pytest_cache
2023-01-11T22:18:17.7053542Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T22:18:17.7054205Z torch: 2.0.0a0+git8419ddd
2023-01-11T22:18:17.7054542Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T22:18:17.7055107Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T22:18:17.7055518Z collecting ... collected 11 items
2023-01-11T22:18:17.7057683Z Running 11 items in this shard: test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None], test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn, test/distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad
2023-01-11T22:18:17.7058993Z 
2023-01-11T22:18:17.7059327Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-1] PASSED [  9%]
2023-01-11T22:18:17.7059897Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[True-4] PASSED [ 18%]
2023-01-11T22:18:17.7060469Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-1] PASSED [ 27%]
2023-01-11T22:18:17.7061051Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_transparency[False-4] PASSED [ 36%]
2023-01-11T22:18:17.7061527Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[0.1] PASSED [ 45%]
2023-01-11T22:18:17.7061997Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_running_stats[None] PASSED [ 54%]
2023-01-11T22:18:17.7062476Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_convert_deferred_batch_norm PASSED [ 63%]
2023-01-11T22:18:17.7062923Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_eval PASSED  [ 72%]
2023-01-11T22:18:17.7063368Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_optimize PASSED [ 81%]
2023-01-11T22:18:17.7063809Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_conv_bn PASSED [ 90%]
2023-01-11T22:18:17.7064264Z distributed/pipeline/sync/test_deferred_batch_norm.py::test_input_requiring_grad PASSED [100%]
2023-01-11T22:18:17.7064514Z 
2023-01-11T22:18:17.7064980Z ============================== 11 passed in 0.59s ==============================
2023-01-11T22:18:17.7065180Z 
2023-01-11T22:18:17.7065500Z ##[endgroup]
2023-01-11T22:18:17.7066193Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_deferred_batch_norm (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_deferred_batch_norm_wjmd9mmf)
2023-01-11T22:18:17.7066609Z 
2023-01-11T22:18:17.7066866Z Running distributed/pipeline/sync/test_bugs ... [2023-01-11 22:18:17.704929]
2023-01-11T22:18:17.7067458Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_bugs.py', '-v'] ... [2023-01-11 22:18:17.705193]
2023-01-11T22:18:24.8845200Z 
2023-01-11T22:18:24.8845912Z Expand the folded group to see the log file of distributed/pipeline/sync/test_bugs
2023-01-11T22:18:24.8847307Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_bugs (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_bugs_hdfvfcim)
2023-01-11T22:18:24.8848123Z ============================= test session starts ==============================
2023-01-11T22:18:24.8848936Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T22:18:24.8849279Z cachedir: .pytest_cache
2023-01-11T22:18:24.8849853Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T22:18:24.8850359Z torch: 2.0.0a0+git8419ddd
2023-01-11T22:18:24.8850877Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T22:18:24.8851460Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T22:18:24.8851852Z collecting ... collected 4 items
2023-01-11T22:18:24.8852523Z Running 4 items in this shard: test/distributed/pipeline/sync/test_bugs.py::test_python_autograd_function, test/distributed/pipeline/sync/test_bugs.py::test_exception_no_hang, test/distributed/pipeline/sync/test_bugs.py::test_tuple_wait, test/distributed/pipeline/sync/test_bugs.py::test_parallel_randoms
2023-01-11T22:18:24.8853048Z 
2023-01-11T22:18:24.8853270Z distributed/pipeline/sync/test_bugs.py::test_python_autograd_function PASSED [ 25%]
2023-01-11T22:18:24.8853696Z distributed/pipeline/sync/test_bugs.py::test_exception_no_hang PASSED    [ 50%]
2023-01-11T22:18:24.8854120Z distributed/pipeline/sync/test_bugs.py::test_tuple_wait PASSED           [ 75%]
2023-01-11T22:18:24.8854538Z distributed/pipeline/sync/test_bugs.py::test_parallel_randoms PASSED     [100%]
2023-01-11T22:18:24.8854784Z 
2023-01-11T22:18:24.8854940Z ============================== 4 passed in 4.66s ===============================
2023-01-11T22:18:24.8855113Z 
2023-01-11T22:18:24.8855435Z ##[endgroup]
2023-01-11T22:18:24.8856049Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_bugs (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_bugs_hdfvfcim)
2023-01-11T22:18:24.8856418Z 
2023-01-11T22:18:24.8856711Z Running distributed/pipeline/sync/skip/test_tracker ... [2023-01-11 22:18:24.884547]
2023-01-11T22:18:24.8857324Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_tracker.py', '-v'] ... [2023-01-11 22:18:24.884803]
2023-01-11T22:18:28.7406174Z 
2023-01-11T22:18:28.7406945Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_tracker
2023-01-11T22:18:28.7408453Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_tracker (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_tracker_hfilg4pl)
2023-01-11T22:18:28.7409018Z ============================= test session starts ==============================
2023-01-11T22:18:28.7409873Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T22:18:28.7410229Z cachedir: .pytest_cache
2023-01-11T22:18:28.7410782Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T22:18:28.7411603Z torch: 2.0.0a0+git8419ddd
2023-01-11T22:18:28.7412185Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T22:18:28.7412758Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T22:18:28.7413152Z collecting ... collected 6 items
2023-01-11T22:18:28.7414118Z Running 6 items in this shard: test/distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker, test/distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel, test/distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal, test/distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal, test/distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing, test/distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing
2023-01-11T22:18:28.7414908Z 
2023-01-11T22:18:28.7415135Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker PASSED [ 16%]
2023-01-11T22:18:28.7415635Z distributed/pipeline/sync/skip/test_tracker.py::test_default_skip_tracker_by_data_parallel PASSED [ 33%]
2023-01-11T22:18:28.7416090Z distributed/pipeline/sync/skip/test_tracker.py::test_reuse_portal PASSED [ 50%]
2023-01-11T22:18:28.7416530Z distributed/pipeline/sync/skip/test_tracker.py::test_no_copy_no_portal PASSED [ 66%]
2023-01-11T22:18:28.7417029Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_without_checkpointing PASSED [ 83%]
2023-01-11T22:18:28.7417525Z distributed/pipeline/sync/skip/test_tracker.py::test_tensor_life_with_checkpointing PASSED [100%]
2023-01-11T22:18:28.7417800Z 
2023-01-11T22:18:28.7417938Z ============================== 6 passed in 1.44s ===============================
2023-01-11T22:18:28.7418129Z 
2023-01-11T22:18:28.7418445Z ##[endgroup]
2023-01-11T22:18:28.7419108Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_tracker (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_tracker_hfilg4pl)
2023-01-11T22:18:28.7419517Z 
2023-01-11T22:18:28.7419784Z Running distributed/pipeline/sync/skip/test_leak ... [2023-01-11 22:18:28.740640]
2023-01-11T22:18:28.7420395Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_leak.py', '-v'] ... [2023-01-11 22:18:28.740894]
2023-01-11T22:18:31.3875506Z 
2023-01-11T22:18:31.3876013Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_leak
2023-01-11T22:18:31.3877283Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_leak (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_leak_x7no6nji)
2023-01-11T22:18:31.3877831Z ============================= test session starts ==============================
2023-01-11T22:18:31.3878435Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T22:18:31.3879050Z cachedir: .pytest_cache
2023-01-11T22:18:31.3879633Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T22:18:31.3880082Z torch: 2.0.0a0+git8419ddd
2023-01-11T22:18:31.3880409Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T22:18:31.3881177Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T22:18:31.3881585Z collecting ... collected 8 items
2023-01-11T22:18:31.3883602Z Running 8 items in this shard: test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[train], test/distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[eval]
2023-01-11T22:18:31.3884769Z 
2023-01-11T22:18:31.3885119Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-train] PASSED [ 12%]
2023-01-11T22:18:31.3885705Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-eval] PASSED [ 25%]
2023-01-11T22:18:31.3886302Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-train] PASSED [ 37%]
2023-01-11T22:18:31.3886923Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-eval] PASSED [ 50%]
2023-01-11T22:18:31.3887493Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-train] PASSED [ 62%]
2023-01-11T22:18:31.3888065Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-eval] PASSED [ 75%]
2023-01-11T22:18:31.3888554Z distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[train] PASSED [ 87%]
2023-01-11T22:18:31.3889010Z distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[eval] PASSED [100%]
2023-01-11T22:18:31.3889271Z 
2023-01-11T22:18:31.3889426Z ============================== 8 passed in 0.27s ===============================
2023-01-11T22:18:31.3889616Z 
2023-01-11T22:18:31.3889930Z ##[endgroup]
2023-01-11T22:18:31.3890558Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_leak (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_leak_x7no6nji)
2023-01-11T22:18:31.3890944Z 
2023-01-11T22:18:31.3891226Z Running distributed/pipeline/sync/skip/test_api ... [2023-01-11 22:18:31.387674]
2023-01-11T22:18:31.3891833Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_api.py', '-v'] ... [2023-01-11 22:18:31.387924]
2023-01-11T22:18:33.7458064Z 
2023-01-11T22:18:33.7459016Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_api
2023-01-11T22:18:33.7460249Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_api (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_api_wtzyqkj7)
2023-01-11T22:18:33.7460817Z ============================= test session starts ==============================
2023-01-11T22:18:33.7461818Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T22:18:33.7462278Z cachedir: .pytest_cache
2023-01-11T22:18:33.7462859Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T22:18:33.7463297Z torch: 2.0.0a0+git8419ddd
2023-01-11T22:18:33.7463755Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T22:18:33.7464413Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T22:18:33.7464828Z collecting ... collected 3 items
2023-01-11T22:18:33.7465441Z Running 3 items in this shard: test/distributed/pipeline/sync/skip/test_api.py::test_namespace_difference, test/distributed/pipeline/sync/skip/test_api.py::test_namespace_copy, test/distributed/pipeline/sync/skip/test_api.py::test_skippable_repr
2023-01-11T22:18:33.7465910Z 
2023-01-11T22:18:33.7466119Z distributed/pipeline/sync/skip/test_api.py::test_namespace_difference PASSED [ 33%]
2023-01-11T22:18:33.7466574Z distributed/pipeline/sync/skip/test_api.py::test_namespace_copy PASSED   [ 66%]
2023-01-11T22:18:33.7467014Z distributed/pipeline/sync/skip/test_api.py::test_skippable_repr PASSED   [100%]
2023-01-11T22:18:33.7467263Z 
2023-01-11T22:18:33.7467422Z ============================== 3 passed in 0.05s ===============================
2023-01-11T22:18:33.7467596Z 
2023-01-11T22:18:33.7467911Z ##[endgroup]
2023-01-11T22:18:33.7468556Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_api (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_api_wtzyqkj7)
2023-01-11T22:18:33.7469272Z 
2023-01-11T22:18:33.7469569Z Running distributed/fsdp/test_shard_utils ... [2023-01-11 22:18:33.745886]
2023-01-11T22:18:33.7470228Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_shard_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:18:33.746155]
2023-01-11T22:18:35.9436427Z 
2023-01-11T22:18:35.9437275Z Expand the folded group to see the log file of distributed/fsdp/test_shard_utils
2023-01-11T22:18:35.9438207Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_shard_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_shard_utils_5ob7py9g)
2023-01-11T22:18:35.9438569Z 
2023-01-11T22:18:35.9438863Z ##[endgroup]
2023-01-11T22:18:35.9439769Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_shard_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_shard_utils_5ob7py9g)
2023-01-11T22:18:35.9440110Z 
2023-01-11T22:18:35.9440579Z Running distributed/_shard/sharded_tensor/ops/test_math_ops ... [2023-01-11 22:18:35.943705]
2023-01-11T22:18:35.9442368Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_math_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:18:35.943975]
2023-01-11T22:18:38.0742952Z 
2023-01-11T22:18:38.0743792Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_math_ops
2023-01-11T22:18:38.0745217Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_math_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_math_ops_sx4cih65)
2023-01-11T22:18:38.0745673Z 
2023-01-11T22:18:38.0745986Z ##[endgroup]
2023-01-11T22:18:38.0746760Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_math_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_math_ops_sx4cih65)
2023-01-11T22:18:38.0747346Z 
2023-01-11T22:18:38.0747841Z Running distributed/checkpoint/test_dedup_tensors ... [2023-01-11 22:18:38.074357]
2023-01-11T22:18:38.0749241Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_dedup_tensors.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:18:38.074617]
2023-01-11T22:18:42.0552437Z 
2023-01-11T22:18:42.0553218Z Expand the folded group to see the log file of distributed/checkpoint/test_dedup_tensors
2023-01-11T22:18:42.0554450Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_dedup_tensors (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_dedup_tensors_cz6bv0xh)
2023-01-11T22:18:42.0554917Z 
2023-01-11T22:18:42.0555055Z Running tests...
2023-01-11T22:18:42.0555568Z ----------------------------------------------------------------------
2023-01-11T22:18:42.0556134Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_dedup_tensors
2023-01-11T22:18:42.0557545Z   test_dedup_shards (__main__.TestDedupTensor) ... 2023-01-11 22:18:41,668 _dedup_tensors.py:44 INFO p:MainProcess t:MainThread: Duplicate keys to remove: {1: [MetadataIndex(fqn='tensor_0', offset=torch.Size([0]), index=None)]}
2023-01-11T22:18:42.0558063Z ok (1.695s)
2023-01-11T22:18:42.0558212Z 
2023-01-11T22:18:42.0558488Z ----------------------------------------------------------------------
2023-01-11T22:18:42.0558792Z Ran 1 test in 1.695s
2023-01-11T22:18:42.0558952Z 
2023-01-11T22:18:42.0559048Z OK
2023-01-11T22:18:42.0559179Z 
2023-01-11T22:18:42.0559303Z Generating XML reports...
2023-01-11T22:18:42.0559893Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_dedup_tensors/TEST-TestDedupTensor-20230111221839.xml
2023-01-11T22:18:42.0560259Z 
2023-01-11T22:18:42.0560570Z ##[endgroup]
2023-01-11T22:18:42.0561202Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_dedup_tensors (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_dedup_tensors_cz6bv0xh)
2023-01-11T22:18:42.0561859Z 
2023-01-11T22:18:42.0562236Z Running distributed/checkpoint/test_nested_dict ... [2023-01-11 22:18:42.055276]
2023-01-11T22:18:42.0562952Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_nested_dict.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:18:42.055548]
2023-01-11T22:18:46.0374877Z 
2023-01-11T22:18:46.0375604Z Expand the folded group to see the log file of distributed/checkpoint/test_nested_dict
2023-01-11T22:18:46.0377308Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_nested_dict (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_nested_dict_fc3f46yu)
2023-01-11T22:18:46.0377683Z 
2023-01-11T22:18:46.0377803Z Running tests...
2023-01-11T22:18:46.0378321Z ----------------------------------------------------------------------
2023-01-11T22:18:46.0378898Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_nested_dict
2023-01-11T22:18:46.0379348Z   test_flattening_round_trip (__main__.TestFlattening) ... ok (1.688s)
2023-01-11T22:18:46.0379749Z   test_mapping (__main__.TestFlattening) ... ok (0.002s)
2023-01-11T22:18:46.0379959Z 
2023-01-11T22:18:46.0380222Z ----------------------------------------------------------------------
2023-01-11T22:18:46.0380551Z Ran 2 tests in 1.690s
2023-01-11T22:18:46.0380694Z 
2023-01-11T22:18:46.0380789Z OK
2023-01-11T22:18:46.0380924Z 
2023-01-11T22:18:46.0381048Z Generating XML reports...
2023-01-11T22:18:46.0381654Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_nested_dict/TEST-TestFlattening-20230111221843.xml
2023-01-11T22:18:46.0382007Z 
2023-01-11T22:18:46.0382298Z ##[endgroup]
2023-01-11T22:18:46.0382922Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_nested_dict (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_nested_dict_fc3f46yu)
2023-01-11T22:18:46.0383297Z 
2023-01-11T22:18:46.0383583Z Running distributed/_composable/test_contract ... [2023-01-11 22:18:46.037489]
2023-01-11T22:18:46.0384284Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_contract.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:18:46.037754]
2023-01-11T22:18:49.9966839Z 
2023-01-11T22:18:49.9967510Z Expand the folded group to see the log file of distributed/_composable/test_contract
2023-01-11T22:18:49.9968574Z ##[group]PRINTING LOG FILE of distributed/_composable/test_contract (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_contract_219klbjm)
2023-01-11T22:18:49.9968959Z 
2023-01-11T22:18:49.9969071Z Running tests...
2023-01-11T22:18:49.9969724Z ----------------------------------------------------------------------
2023-01-11T22:18:49.9970476Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_contract
2023-01-11T22:18:49.9970892Z   test_add_hooks (__main__.TestContract) ... ok (1.648s)
2023-01-11T22:18:49.9971244Z   test_modify_fqn (__main__.TestContract) ... ok (0.002s)
2023-01-11T22:18:49.9971615Z   test_registry (__main__.TestContract) ... ok (0.002s)
2023-01-11T22:18:49.9972009Z   test_state (__main__.TestContract) ... ok (0.002s)
2023-01-11T22:18:49.9972376Z 
2023-01-11T22:18:49.9972688Z ----------------------------------------------------------------------
2023-01-11T22:18:49.9973021Z Ran 4 tests in 1.655s
2023-01-11T22:18:49.9973184Z 
2023-01-11T22:18:49.9973260Z OK
2023-01-11T22:18:49.9973396Z 
2023-01-11T22:18:49.9973523Z Generating XML reports...
2023-01-11T22:18:49.9974109Z Generated XML report: test-reports/python-unittest/distributed._composable.test_contract/TEST-TestContract-20230111221847.xml
2023-01-11T22:18:49.9974449Z 
2023-01-11T22:18:49.9974741Z ##[endgroup]
2023-01-11T22:18:49.9975349Z FINISHED PRINTING LOG FILE of distributed/_composable/test_contract (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_contract_219klbjm)
2023-01-11T22:18:49.9975706Z 
2023-01-11T22:18:49.9975966Z Running distributed/fsdp/test_utils ... [2023-01-11 22:18:49.996729]
2023-01-11T22:18:49.9976997Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:18:49.996993]
2023-01-11T22:18:54.0029631Z 
2023-01-11T22:18:54.0030276Z Expand the folded group to see the log file of distributed/fsdp/test_utils
2023-01-11T22:18:54.0031527Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_utils_ophem5qq)
2023-01-11T22:18:54.0032129Z 
2023-01-11T22:18:54.0032242Z Running tests...
2023-01-11T22:18:54.0032748Z ----------------------------------------------------------------------
2023-01-11T22:18:54.0033289Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_utils
2023-01-11T22:18:54.0034205Z   test_get_fully_sharded_module_to_states (__main__.TestGetSubmoduleToStates)
2023-01-11T22:18:54.0034684Z Tests the helper function ``_get_fully_sharded_module_states()`` that ... ok (1.683s)
2023-01-11T22:18:54.0035116Z   test_apply_to_tensors_cpu_cuda (__main__.TestUtils) ... ok (0.003s)
2023-01-11T22:18:54.0035605Z   test_apply_to_tensors_devices_['cpu'] (__main__.TestUtils) ... ok (0.003s)
2023-01-11T22:18:54.0036059Z   test_apply_to_tensors_devices_['cuda'] (__main__.TestUtils) ... ok (0.003s)
2023-01-11T22:18:54.0036407Z   test_packed_sequence (__main__.TestUtils)
2023-01-11T22:18:54.0036769Z Test to ensure RNN packed sequences are modified correctly. ... ok (0.002s)
2023-01-11T22:18:54.0037139Z   test_replace_by_prefix (__main__.TestUtils) ... ok (0.001s)
2023-01-11T22:18:54.0037353Z 
2023-01-11T22:18:54.0037619Z ----------------------------------------------------------------------
2023-01-11T22:18:54.0037948Z Ran 6 tests in 1.697s
2023-01-11T22:18:54.0038110Z 
2023-01-11T22:18:54.0038201Z OK
2023-01-11T22:18:54.0038347Z 
2023-01-11T22:18:54.0038454Z Generating XML reports...
2023-01-11T22:18:54.0039064Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestGetSubmoduleToStates-20230111221851.xml
2023-01-11T22:18:54.0039804Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20230111221851.xml
2023-01-11T22:18:54.0040123Z 
2023-01-11T22:18:54.0040429Z ##[endgroup]
2023-01-11T22:18:54.0040989Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_utils_ophem5qq)
2023-01-11T22:18:54.0041324Z 
2023-01-11T22:18:54.0041578Z Running distributed/fsdp/test_fsdp_fx ... [2023-01-11 22:18:54.002950]
2023-01-11T22:18:54.0042231Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fx.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:18:54.003243]
2023-01-11T22:18:58.0007643Z 
2023-01-11T22:18:58.0008397Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_fx
2023-01-11T22:18:58.0009413Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_fx (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_fx_xyab3su9)
2023-01-11T22:18:58.0009803Z 
2023-01-11T22:18:58.0009927Z Running tests...
2023-01-11T22:18:58.0010473Z ----------------------------------------------------------------------
2023-01-11T22:18:58.0011196Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_fx
2023-01-11T22:18:58.0011644Z   test_symbolic_tracing_outputs (__main__.TestSymbolicTracing)
2023-01-11T22:18:58.0012327Z Tests running ``tracer.trace()`` inside ``patch_tracer()`` by checking ... ok (1.689s)
2023-01-11T22:18:58.0012579Z 
2023-01-11T22:18:58.0012851Z ----------------------------------------------------------------------
2023-01-11T22:18:58.0013300Z Ran 1 test in 1.689s
2023-01-11T22:18:58.0013528Z 
2023-01-11T22:18:58.0013623Z OK
2023-01-11T22:18:58.0013738Z 
2023-01-11T22:18:58.0013862Z Generating XML reports...
2023-01-11T22:18:58.0014645Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20230111221855.xml
2023-01-11T22:18:58.0015332Z 
2023-01-11T22:18:58.0015652Z ##[endgroup]
2023-01-11T22:18:58.0016321Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_fx (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_fx_xyab3su9)
2023-01-11T22:18:58.0016675Z 
2023-01-11T22:18:58.0016956Z Running distributed/elastic/utils/util_test ... [2023-01-11 22:18:58.000810]
2023-01-11T22:18:58.0017630Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/util_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:18:58.001114]
2023-01-11T22:19:02.0583921Z 
2023-01-11T22:19:02.0584706Z Expand the folded group to see the log file of distributed/elastic/utils/util_test
2023-01-11T22:19:02.0585636Z ##[group]PRINTING LOG FILE of distributed/elastic/utils/util_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-util_test_o0eqn9yv)
2023-01-11T22:19:02.0586019Z 
2023-01-11T22:19:02.0586136Z Running tests...
2023-01-11T22:19:02.0586893Z ----------------------------------------------------------------------
2023-01-11T22:19:02.0587507Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.util_test
2023-01-11T22:19:02.0587932Z   test_get_all_rank_0 (__main__.StoreUtilTest) ... ok (1.685s)
2023-01-11T22:19:02.0588286Z   test_get_all_rank_n (__main__.StoreUtilTest) ... ok (0.002s)
2023-01-11T22:19:02.0588636Z   test_synchronize (__main__.StoreUtilTest) ... ok (0.003s)
2023-01-11T22:19:02.0588962Z   test_get_logger (__main__.UtilTest) ... ok (0.107s)
2023-01-11T22:19:02.0589608Z   test_get_logger_custom_name (__main__.UtilTest) ... ok (0.001s)
2023-01-11T22:19:02.0590133Z   test_get_logger_different (__main__.UtilTest) ... ok (0.001s)
2023-01-11T22:19:02.0590483Z   test_get_logger_none (__main__.UtilTest) ... ok (0.001s)
2023-01-11T22:19:02.0590666Z 
2023-01-11T22:19:02.0590951Z ----------------------------------------------------------------------
2023-01-11T22:19:02.0591276Z Ran 7 tests in 1.801s
2023-01-11T22:19:02.0591446Z 
2023-01-11T22:19:02.0591540Z OK
2023-01-11T22:19:02.0591669Z 
2023-01-11T22:19:02.0591781Z Generating XML reports...
2023-01-11T22:19:02.0592756Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20230111221859.xml
2023-01-11T22:19:02.0593502Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20230111221859.xml
2023-01-11T22:19:02.0593831Z 
2023-01-11T22:19:02.0594138Z ##[endgroup]
2023-01-11T22:19:02.0594746Z FINISHED PRINTING LOG FILE of distributed/elastic/utils/util_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-util_test_o0eqn9yv)
2023-01-11T22:19:02.0595110Z 
2023-01-11T22:19:02.0595388Z Running distributed/rpc/test_share_memory ... [2023-01-11 22:19:02.058488]
2023-01-11T22:19:02.0596094Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_share_memory.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:19:02.058808]
2023-01-11T22:19:10.7778210Z 
2023-01-11T22:19:10.7779074Z Expand the folded group to see the log file of distributed/rpc/test_share_memory
2023-01-11T22:19:10.7780997Z ##[group]PRINTING LOG FILE of distributed/rpc/test_share_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_share_memory_wtjwgevr)
2023-01-11T22:19:10.7781654Z <unittest.suite.TestSuite tests=[]>
2023-01-11T22:19:10.7782373Z <unittest.suite.TestSuite tests=[<__main__.TestRPCPickler testMethod=test_case>]>
2023-01-11T22:19:10.7783042Z test_case (__main__.TestRPCPickler)
2023-01-11T22:19:10.7783720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:19:10.7784542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:19:10.7785623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:19:10.7786335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:19:10.7787163Z 
2023-01-11T22:19:10.7787337Z Running tests...
2023-01-11T22:19:10.7787977Z ----------------------------------------------------------------------
2023-01-11T22:19:10.7788838Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_share_memory
2023-01-11T22:19:10.7789925Z   test_case (__main__.TestRPCPickler) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:19:10.7790683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:19:10.7791580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:19:10.7792958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:19:10.7794612Z /opt/conda/lib/python3.10/site-packages/torch/multiprocessing/reductions.py:355: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:19:10.7795333Z   if storage.is_cuda:
2023-01-11T22:19:10.7795573Z ok (4.353s)
2023-01-11T22:19:10.7795719Z 
2023-01-11T22:19:10.7795996Z ----------------------------------------------------------------------
2023-01-11T22:19:10.7796305Z Ran 1 test in 4.353s
2023-01-11T22:19:10.7796465Z 
2023-01-11T22:19:10.7796557Z OK
2023-01-11T22:19:10.7796692Z 
2023-01-11T22:19:10.7796816Z Generating XML reports...
2023-01-11T22:19:10.7797381Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_share_memory/TEST-TestRPCPickler-20230111221905.xml
2023-01-11T22:19:10.7797724Z 
2023-01-11T22:19:10.7798079Z ##[endgroup]
2023-01-11T22:19:10.7798670Z FINISHED PRINTING LOG FILE of distributed/rpc/test_share_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_share_memory_wtjwgevr)
2023-01-11T22:19:10.7799030Z 
2023-01-11T22:19:10.7799295Z Running distributed/test_data_parallel ... [2023-01-11 22:19:10.777945]
2023-01-11T22:19:10.7799946Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_data_parallel.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:19:10.778282]
2023-01-11T22:19:19.6708800Z 
2023-01-11T22:19:19.6709330Z Expand the folded group to see the log file of distributed/test_data_parallel
2023-01-11T22:19:19.6710251Z ##[group]PRINTING LOG FILE of distributed/test_data_parallel (/var/lib/jenkins/workspace/test/test-reports/distributed-test_data_parallel_0lied_c5)
2023-01-11T22:19:19.6710601Z 
2023-01-11T22:19:19.6710713Z Running tests...
2023-01-11T22:19:19.6711210Z ----------------------------------------------------------------------
2023-01-11T22:19:19.6711751Z Test results will be stored in test-reports/python-unittest/distributed.test_data_parallel
2023-01-11T22:19:19.6712482Z   test_autocast (__main__.TestDataParallel) ... ok (1.907s)
2023-01-11T22:19:19.6712868Z   test_data_parallel (__main__.TestDataParallel) ... ok (0.093s)
2023-01-11T22:19:19.6713288Z   test_data_parallel_buffers_requiring_grad (__main__.TestDataParallel) ... ok (0.013s)
2023-01-11T22:19:19.6713713Z   test_data_parallel_complex (__main__.TestDataParallel) ... ok (0.066s)
2023-01-11T22:19:19.6714122Z   test_data_parallel_device_args (__main__.TestDataParallel) ... ok (0.006s)
2023-01-11T22:19:19.6714529Z   test_data_parallel_function_deletion (__main__.TestDataParallel) ... ok (0.006s)
2023-01-11T22:19:19.6715446Z   test_data_parallel_lazy_linear (__main__.TestDataParallel) ... /opt/conda/lib/python3.10/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment.
2023-01-11T22:19:19.6716201Z   warnings.warn('Lazy modules are a new feature under heavy development '
2023-01-11T22:19:19.6716524Z ok (0.002s)
2023-01-11T22:19:19.6717196Z   test_data_parallel_model_device (__main__.TestDataParallel)
2023-01-11T22:19:19.6717591Z Test device[0] check at forward time. ... ok (0.033s)
2023-01-11T22:19:19.6718007Z   test_data_parallel_model_no_refcycles (__main__.TestDataParallel) ... ok (0.093s)
2023-01-11T22:19:19.6718469Z   test_data_parallel_module_zero_inputs (__main__.TestDataParallel) ... ok (0.004s)
2023-01-11T22:19:19.6719391Z   test_data_parallel_multiple_input (__main__.TestDataParallel) ... /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/comm.py:231: UserWarning: Using -1 to represent CPU tensor is deprecated. Please use a device object or string instead, e.g., "cpu".
2023-01-11T22:19:19.6719984Z   warnings.warn(
2023-01-11T22:19:19.6720225Z ok (0.022s)
2023-01-11T22:19:19.6720552Z   test_data_parallel_nested_input (__main__.TestDataParallel) ... ok (0.003s)
2023-01-11T22:19:19.6720994Z   test_data_parallel_nested_output (__main__.TestDataParallel) ... ok (0.006s)
2023-01-11T22:19:19.6721423Z   test_data_parallel_no_grad (__main__.TestDataParallel) ... ok (0.004s)
2023-01-11T22:19:19.6721850Z   test_data_parallel_rnn (__main__.TestDataParallel) ... ok (1.066s)
2023-01-11T22:19:19.6722248Z   test_data_parallel_small_back (__main__.TestDataParallel) ... ok (0.003s)
2023-01-11T22:19:19.6722669Z   test_data_parallel_sparse (__main__.TestDataParallel) ... ok (0.012s)
2023-01-11T22:19:19.6723571Z   test_gather_cpu (__main__.TestDataParallel) ... /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
2023-01-11T22:19:19.6724305Z   warnings.warn('Was asked to gather along dimension 0, but all '
2023-01-11T22:19:19.6724605Z ok (0.044s)
2023-01-11T22:19:19.6724944Z   test_gather_different_len_dicts (__main__.TestDataParallel) ... ok (0.001s)
2023-01-11T22:19:19.6725353Z   test_gather_gpu (__main__.TestDataParallel) ... ok (0.043s)
2023-01-11T22:19:19.6725735Z   test_parallel_apply (__main__.TestDataParallel) ... ok (0.005s)
2023-01-11T22:19:19.6726157Z   test_parallel_apply_autocast (__main__.TestDataParallel) ... ok (0.005s)
2023-01-11T22:19:19.6726607Z   test_parallel_apply_passes_exception (__main__.TestDataParallel) ... ok (0.002s)
2023-01-11T22:19:19.6727036Z   test_parameter_list_dict_replica (__main__.TestDataParallel) ... ok (0.007s)
2023-01-11T22:19:19.6727450Z   test_replicate (__main__.TestDataParallel) ... ok (0.004s)
2023-01-11T22:19:19.6727851Z   test_replicate_buffers (__main__.TestDataParallel) ... ok (0.003s)
2023-01-11T22:19:19.6728266Z   test_save_replica_module (__main__.TestDataParallel) ... ok (0.003s)
2023-01-11T22:19:19.6728652Z   test_scatter_cpu (__main__.TestDataParallel) ... ok (0.018s)
2023-01-11T22:19:19.6729036Z   test_scatter_gpu (__main__.TestDataParallel) ... ok (0.018s)
2023-01-11T22:19:19.6729435Z   test_strided_grad_layout (__main__.TestDataParallel) ... ok (1.015s)
2023-01-11T22:19:19.6729808Z   test_zero_grad (__main__.TestDataParallel) ... ok (0.008s)
2023-01-11T22:19:19.6730274Z   test_data_parallel_module_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2023-01-11T22:19:19.6730807Z   test_data_parallel_module_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2023-01-11T22:19:19.6731343Z   test_data_parallel_module_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2023-01-11T22:19:19.6731904Z   test_data_parallel_module_kwargs_only_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2023-01-11T22:19:19.6732467Z   test_data_parallel_module_kwargs_only_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2023-01-11T22:19:19.6733031Z   test_data_parallel_module_kwargs_only_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2023-01-11T22:19:19.6733617Z   test_data_parallel_module_kwargs_only_empty_dict_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s)
2023-01-11T22:19:19.6734265Z   test_data_parallel_module_kwargs_only_empty_dict_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s)
2023-01-11T22:19:19.6734923Z   test_data_parallel_module_kwargs_only_empty_dict_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2023-01-11T22:19:19.6735517Z   test_data_parallel_module_kwargs_only_empty_list_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2023-01-11T22:19:19.6736161Z   test_data_parallel_module_kwargs_only_empty_list_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s)
2023-01-11T22:19:19.6751175Z   test_data_parallel_module_kwargs_only_empty_list_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.004s)
2023-01-11T22:19:19.6751767Z   test_data_parallel_module_kwargs_only_empty_tuple_cuda_float16 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s)
2023-01-11T22:19:19.6752577Z   test_data_parallel_module_kwargs_only_empty_tuple_cuda_float32 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s)
2023-01-11T22:19:19.6753155Z   test_data_parallel_module_kwargs_only_empty_tuple_cuda_float64 (__main__.TestDataParallelDeviceTypeCUDA) ... ok (0.005s)
2023-01-11T22:19:19.6753460Z 
2023-01-11T22:19:19.6753776Z ----------------------------------------------------------------------
2023-01-11T22:19:19.6754112Z Ran 46 tests in 4.588s
2023-01-11T22:19:19.6754257Z 
2023-01-11T22:19:19.6754351Z OK
2023-01-11T22:19:19.6754484Z 
2023-01-11T22:19:19.6754609Z Generating XML reports...
2023-01-11T22:19:19.6755194Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20230111221914.xml
2023-01-11T22:19:19.6755977Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCUDA-20230111221914.xml
2023-01-11T22:19:19.6756366Z 
2023-01-11T22:19:19.6756712Z ##[endgroup]
2023-01-11T22:19:19.6757292Z FINISHED PRINTING LOG FILE of distributed/test_data_parallel (/var/lib/jenkins/workspace/test/test-reports/distributed-test_data_parallel_0lied_c5)
2023-01-11T22:19:19.6757642Z 
2023-01-11T22:19:19.6757916Z Running distributed/fsdp/test_fsdp_multiple_forward ... [2023-01-11 22:19:19.670934]
2023-01-11T22:19:19.6758629Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_forward.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:19:19.671224]
2023-01-11T22:19:27.5012603Z 
2023-01-11T22:19:27.5013300Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_multiple_forward
2023-01-11T22:19:27.5014316Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_forward (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_forward_dhln8adb)
2023-01-11T22:19:27.5014712Z 
2023-01-11T22:19:27.5014831Z Running tests...
2023-01-11T22:19:27.5015386Z ----------------------------------------------------------------------
2023-01-11T22:19:27.5015947Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward
2023-01-11T22:19:27.5016488Z   test_multi_forward (__main__.TestMultiForward) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:19:27.5016957Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51777
2023-01-11T22:19:27.5017407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51778
2023-01-11T22:19:27.5018011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:19:27.5018468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:19:27.5019042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:19:27.5019516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:19:27.5020072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:19:27.5020518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:19:27.5021459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:19:27.5021916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:19:27.5022372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:19:27.5022869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:19:27.5023530Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:19:27.5024203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:19:27.5024731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:19:27.5025207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:19:27.5025692Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:19:27.5026158Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:19:27.5027437Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:19:27.5028226Z   warnings.warn(
2023-01-11T22:19:27.5029377Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:19:27.5030157Z   warnings.warn(
2023-01-11T22:19:27.5030389Z dist init r=0, world=2
2023-01-11T22:19:27.5030642Z dist init r=1, world=2
2023-01-11T22:19:27.5030884Z ok (5.479s)
2023-01-11T22:19:27.5031033Z 
2023-01-11T22:19:27.5031285Z ----------------------------------------------------------------------
2023-01-11T22:19:27.5031618Z Ran 1 test in 5.479s
2023-01-11T22:19:27.5031778Z 
2023-01-11T22:19:27.5032218Z OK
2023-01-11T22:19:27.5032371Z 
2023-01-11T22:19:27.5032499Z Generating XML reports...
2023-01-11T22:19:27.5033112Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20230111221921.xml
2023-01-11T22:19:27.5033483Z 
2023-01-11T22:19:27.5033801Z ##[endgroup]
2023-01-11T22:19:27.5034448Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_forward (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_forward_dhln8adb)
2023-01-11T22:19:27.5034806Z 
2023-01-11T22:19:27.5035114Z Running distributed/fsdp/test_fsdp_multiple_wrapping ... [2023-01-11 22:19:27.501252]
2023-01-11T22:19:27.5035832Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_wrapping.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:19:27.501506]
2023-01-11T22:19:35.1570743Z 
2023-01-11T22:19:35.1571303Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_multiple_wrapping
2023-01-11T22:19:35.1572325Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_wrapping (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_wrapping_3rco9x_2)
2023-01-11T22:19:35.1572722Z 
2023-01-11T22:19:35.1573124Z Running tests...
2023-01-11T22:19:35.1573727Z ----------------------------------------------------------------------
2023-01-11T22:19:35.1574332Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping
2023-01-11T22:19:35.1574797Z   test_multiple_wrapping (__main__.TestMultipleWrapping)
2023-01-11T22:19:35.1575243Z This test simulates wrapping the module after training to run inference. ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:19:35.1575734Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51895
2023-01-11T22:19:35.1576215Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51896
2023-01-11T22:19:35.1576846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:19:35.1577277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:19:35.1577850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:19:35.1578325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:19:35.1578904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:19:35.1579327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:19:35.1579892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:19:35.1580350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:19:35.1580783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:19:35.1581276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:19:35.1581927Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:19:35.1582624Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:19:35.1583125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:19:35.1583590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:19:35.1584841Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:19:35.1585614Z   warnings.warn(
2023-01-11T22:19:35.1586769Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:19:35.1587552Z   warnings.warn(
2023-01-11T22:19:35.1587788Z dist init r=0, world=2
2023-01-11T22:19:35.1588031Z dist init r=1, world=2
2023-01-11T22:19:35.1588266Z ok (5.334s)
2023-01-11T22:19:35.1588414Z 
2023-01-11T22:19:35.1588664Z ----------------------------------------------------------------------
2023-01-11T22:19:35.1588998Z Ran 1 test in 5.334s
2023-01-11T22:19:35.1589158Z 
2023-01-11T22:19:35.1589250Z OK
2023-01-11T22:19:35.1589386Z 
2023-01-11T22:19:35.1589492Z Generating XML reports...
2023-01-11T22:19:35.1590202Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20230111221929.xml
2023-01-11T22:19:35.1590638Z 
2023-01-11T22:19:35.1590958Z ##[endgroup]
2023-01-11T22:19:35.1591595Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_wrapping (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_wrapping_3rco9x_2)
2023-01-11T22:19:35.1592299Z 
2023-01-11T22:19:35.1592632Z Running distributed/_shard/sharded_optim/test_sharded_optim ... [2023-01-11 22:19:35.157099]
2023-01-11T22:19:35.1593389Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_optim/test_sharded_optim.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:19:35.157355]
2023-01-11T22:19:43.9421875Z 
2023-01-11T22:19:43.9422650Z Expand the folded group to see the log file of distributed/_shard/sharded_optim/test_sharded_optim
2023-01-11T22:19:43.9423647Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_optim/test_sharded_optim (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_optim-test_sharded_optim_1rj2jqeg)
2023-01-11T22:19:43.9424062Z 
2023-01-11T22:19:43.9424180Z Running tests...
2023-01-11T22:19:43.9424719Z ----------------------------------------------------------------------
2023-01-11T22:19:43.9425307Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim
2023-01-11T22:19:43.9425839Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:19:43.9426337Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52013
2023-01-11T22:19:43.9426786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52014
2023-01-11T22:19:43.9427224Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 52015
2023-01-11T22:19:43.9427642Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 52016
2023-01-11T22:19:43.9428265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:19:43.9428718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:19:43.9429266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:19:43.9429736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:19:43.9430309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:19:43.9430746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:19:43.9431293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:19:43.9431754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:19:43.9432625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:19:43.9433070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:19:43.9433619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:19:43.9434078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:19:43.9434652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:19:43.9435071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:19:43.9435634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:19:43.9436089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:19:43.9436523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:19:43.9437290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:19:43.9437798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:19:43.9438266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:19:43.9438636Z skip: Need at least 4 CUDA devices (4.074s)
2023-01-11T22:19:43.9439108Z   test_sharded_optim (__main__.TestShardedOptimizer) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52149
2023-01-11T22:19:43.9439627Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52150
2023-01-11T22:19:43.9440067Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 52151
2023-01-11T22:19:43.9440489Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 52152
2023-01-11T22:19:43.9441091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:19:43.9441568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:19:43.9442139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:19:43.9442588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:19:43.9443157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:19:43.9443597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:19:43.9444158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:19:43.9444602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:19:43.9445167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:19:43.9445608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:19:43.9446151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:19:43.9446610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:19:43.9447173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:19:43.9447605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:19:43.9448149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:19:43.9448607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:19:43.9449041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:19:43.9449492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:19:43.9449963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:19:43.9450419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:19:43.9450808Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:19:43.9450982Z 
2023-01-11T22:19:43.9451255Z ----------------------------------------------------------------------
2023-01-11T22:19:43.9451587Z Ran 2 tests in 6.485s
2023-01-11T22:19:43.9451748Z 
2023-01-11T22:19:43.9451856Z OK (skipped=2)
2023-01-11T22:19:43.9452008Z 
2023-01-11T22:19:43.9452113Z Generating XML reports...
2023-01-11T22:19:43.9452749Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20230111221937.xml
2023-01-11T22:19:43.9453127Z 
2023-01-11T22:19:43.9453448Z ##[endgroup]
2023-01-11T22:19:43.9454161Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_optim/test_sharded_optim (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_optim-test_sharded_optim_1rj2jqeg)
2023-01-11T22:19:43.9454608Z 
2023-01-11T22:19:43.9454933Z Running distributed/_shard/sharded_tensor/test_sharded_tensor_reshard ... [2023-01-11 22:19:43.942229]
2023-01-11T22:19:43.9455696Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:19:43.942479]
2023-01-11T22:19:52.6822806Z 
2023-01-11T22:19:52.6823623Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/test_sharded_tensor_reshard
2023-01-11T22:19:52.6824688Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor_reshard (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_reshard_qvwtgzzf)
2023-01-11T22:19:52.6825123Z 
2023-01-11T22:19:52.6825242Z Running tests...
2023-01-11T22:19:52.6826080Z ----------------------------------------------------------------------
2023-01-11T22:19:52.6826721Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard
2023-01-11T22:19:52.6827232Z   test_sharded_tensor_reshard (__main__.TestReshard) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:19:52.6827694Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52320
2023-01-11T22:19:52.6828393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52321
2023-01-11T22:19:52.6828824Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 52322
2023-01-11T22:19:52.6829273Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 52323
2023-01-11T22:19:52.6829900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:19:52.6830358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:19:52.6830928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:19:52.6831399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:19:52.6832321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:19:52.6832787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:19:52.6833334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:19:52.6833784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:19:52.6834353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:19:52.6834804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:19:52.6835397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:19:52.6835859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:19:52.6836436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:19:52.6836859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:19:52.6837468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:19:52.6837932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:19:52.6838415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:19:52.6838871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:19:52.6839330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:19:52.6840155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:19:52.6840542Z skip: Need at least 4 CUDA devices (4.058s)
2023-01-11T22:19:52.6841025Z   test_sharded_tensor_reshard_errors (__main__.TestReshard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52456
2023-01-11T22:19:52.6841545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52457
2023-01-11T22:19:52.6841992Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 52458
2023-01-11T22:19:52.6842408Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 52459
2023-01-11T22:19:52.6843020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:19:52.6843471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:19:52.6844052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:19:52.6844509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:19:52.6845082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:19:52.6845522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:19:52.6846074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:19:52.6846534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:19:52.6847102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:19:52.6847540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:19:52.6848086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:19:52.6848552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:19:52.6849120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:19:52.6849535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:19:52.6850100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:19:52.6850558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:19:52.6850990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:19:52.6851444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:19:52.6851900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:19:52.6852367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:19:52.6852755Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:19:52.6852932Z 
2023-01-11T22:19:52.6853209Z ----------------------------------------------------------------------
2023-01-11T22:19:52.6853540Z Ran 2 tests in 6.468s
2023-01-11T22:19:52.6853702Z 
2023-01-11T22:19:52.6853812Z OK (skipped=2)
2023-01-11T22:19:52.6853966Z 
2023-01-11T22:19:52.6854071Z Generating XML reports...
2023-01-11T22:19:52.6854692Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20230111221945.xml
2023-01-11T22:19:52.6855056Z 
2023-01-11T22:19:52.6855389Z ##[endgroup]
2023-01-11T22:19:52.6856075Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor_reshard (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_reshard_qvwtgzzf)
2023-01-11T22:19:52.6856582Z 
2023-01-11T22:19:52.6856930Z Running distributed/fsdp/test_distributed_checkpoint ... [2023-01-11 22:19:52.682339]
2023-01-11T22:19:52.6857659Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_distributed_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:19:52.682586]
2023-01-11T22:20:03.4387879Z 
2023-01-11T22:20:03.4388582Z Expand the folded group to see the log file of distributed/fsdp/test_distributed_checkpoint
2023-01-11T22:20:03.4389776Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_distributed_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_distributed_checkpoint_sot8moiz)
2023-01-11T22:20:03.4390209Z 
2023-01-11T22:20:03.4390319Z Running tests...
2023-01-11T22:20:03.4390835Z ----------------------------------------------------------------------
2023-01-11T22:20:03.4391400Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint
2023-01-11T22:20:03.4392385Z   test_distributed_checkpoint_state_dict_type_StateDictType_LOCAL_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:03.4392952Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52627
2023-01-11T22:20:03.4393649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52628
2023-01-11T22:20:03.4394264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:03.4394712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:03.4395279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:03.4395743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:03.4396299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:03.4396747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:03.4397311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:03.4397755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:03.4398205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:20:03.4398693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:20:03.4399341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:20:03.4400005Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:20:03.4400518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:20:03.4400989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:20:03.4401339Z dist init r=1, world=2
2023-01-11T22:20:03.4401567Z dist init r=0, world=2
2023-01-11T22:20:03.4401801Z ok (4.987s)
2023-01-11T22:20:03.4402319Z   test_distributed_checkpoint_state_dict_type_StateDictType_SHARDED_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52710
2023-01-11T22:20:03.4402903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52711
2023-01-11T22:20:03.4403506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:03.4403949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:03.4404515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:03.4405236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:03.4405900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:03.4406350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:03.4406898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:03.4407382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:03.4407828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:20:03.4408312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:20:03.4408946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:20:03.4409635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:20:03.4410150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:20:03.4410611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:20:03.4410946Z dist init r=0, world=2
2023-01-11T22:20:03.4411191Z dist init r=1, world=2
2023-01-11T22:20:03.4411427Z ok (3.411s)
2023-01-11T22:20:03.4411576Z 
2023-01-11T22:20:03.4411827Z ----------------------------------------------------------------------
2023-01-11T22:20:03.4412151Z Ran 2 tests in 8.398s
2023-01-11T22:20:03.4412309Z 
2023-01-11T22:20:03.4412400Z OK
2023-01-11T22:20:03.4412531Z 
2023-01-11T22:20:03.4412636Z Generating XML reports...
2023-01-11T22:20:03.4413287Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20230111221954.xml
2023-01-11T22:20:03.4413678Z 
2023-01-11T22:20:03.4414007Z ##[endgroup]
2023-01-11T22:20:03.4414659Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_distributed_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_distributed_checkpoint_sot8moiz)
2023-01-11T22:20:03.4415026Z 
2023-01-11T22:20:03.4415331Z Running distributed/_shard/sharding_spec/test_sharding_spec ... [2023-01-11 22:20:03.438835]
2023-01-11T22:20:03.4416063Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:20:03.439265]
2023-01-11T22:20:14.5664579Z 
2023-01-11T22:20:14.5665555Z Expand the folded group to see the log file of distributed/_shard/sharding_spec/test_sharding_spec
2023-01-11T22:20:14.5666699Z ##[group]PRINTING LOG FILE of distributed/_shard/sharding_spec/test_sharding_spec (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_spec-test_sharding_spec_woxh7w34)
2023-01-11T22:20:14.5667352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2htef3oc
2023-01-11T22:20:14.5668236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2htef3oc/_remote_module_non_scriptable.py
2023-01-11T22:20:14.5668597Z 
2023-01-11T22:20:14.5668711Z Running tests...
2023-01-11T22:20:14.5669423Z ----------------------------------------------------------------------
2023-01-11T22:20:14.5670060Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec
2023-01-11T22:20:14.5670871Z   test_custom_sharding_spec (__main__.TestCustomShardingSpec) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:14.5671519Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52828
2023-01-11T22:20:14.5672237Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52829
2023-01-11T22:20:14.5672670Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 52830
2023-01-11T22:20:14.5673363Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 52831
2023-01-11T22:20:14.5674129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:14.5674585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:14.5675141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:14.5675615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:14.5676199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:14.5676629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:14.5677206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:14.5677672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:14.5678262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:14.5678688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:14.5679260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:14.5679731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:14.5680288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:14.5680734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:14.5681299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:14.5681764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:14.5682217Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6joe55xt
2023-01-11T22:20:14.5682769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6joe55xt/_remote_module_non_scriptable.py
2023-01-11T22:20:14.5683307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb00kx1k5
2023-01-11T22:20:14.5683851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb00kx1k5/_remote_module_non_scriptable.py
2023-01-11T22:20:14.5684366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuuuontoa
2023-01-11T22:20:14.5684905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuuuontoa/_remote_module_non_scriptable.py
2023-01-11T22:20:14.5685437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ey3zx5w
2023-01-11T22:20:14.5685954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ey3zx5w/_remote_module_non_scriptable.py
2023-01-11T22:20:14.5686469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:20:14.5686945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:20:14.5687417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:20:14.5687871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:20:14.5688211Z ok (3.979s)
2023-01-11T22:20:14.5688549Z   test_custom_sharding_spec_shard_tensor (__main__.TestCustomShardingSpec)
2023-01-11T22:20:14.5689035Z Test custom spec can be invoked from the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52964
2023-01-11T22:20:14.5689539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52965
2023-01-11T22:20:14.5689991Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 52966
2023-01-11T22:20:14.5690439Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 52967
2023-01-11T22:20:14.5691163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:14.5691620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:14.5692198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:14.5692646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:14.5693223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:14.5693665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:14.5694239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:14.5694683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:14.5695271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:14.5695719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:14.5696285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:14.5696764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:14.5697342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:14.5697784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:14.5698334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:14.5698796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:14.5699316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkyb8mvi9
2023-01-11T22:20:14.5699873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkyb8mvi9/_remote_module_non_scriptable.py
2023-01-11T22:20:14.5700414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoea0lknb
2023-01-11T22:20:14.5700938Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoea0lknb/_remote_module_non_scriptable.py
2023-01-11T22:20:14.5701479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq2m3s34b
2023-01-11T22:20:14.5702023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq2m3s34b/_remote_module_non_scriptable.py
2023-01-11T22:20:14.5702519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:20:14.5703022Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcbijpbca
2023-01-11T22:20:14.5703521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:20:14.5704043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcbijpbca/_remote_module_non_scriptable.py
2023-01-11T22:20:14.5704537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:20:14.5705012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:20:14.5705405Z skip: Need at least 4 CUDA devices (2.409s)
2023-01-11T22:20:14.5705784Z   test_custom_sharding_spec_tensor_ctor (__main__.TestCustomShardingSpec)
2023-01-11T22:20:14.5706280Z Test sharded_tensor.ones(...) with the custom ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53100
2023-01-11T22:20:14.5706792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53101
2023-01-11T22:20:14.5707240Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53102
2023-01-11T22:20:14.5707666Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53103
2023-01-11T22:20:14.5708406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:14.5708862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:14.5709441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:14.5709895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:14.5710473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:14.5710925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:14.5711477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:14.5712254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:14.5712895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:14.5713342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:14.5713892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:14.5714357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:14.5714932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:14.5715376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:14.5715924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:14.5716387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:14.5716859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnj1nz94m
2023-01-11T22:20:14.5717397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnj1nz94m/_remote_module_non_scriptable.py
2023-01-11T22:20:14.5717913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:20:14.5718416Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu2qmiy6j
2023-01-11T22:20:14.5718965Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu2qmiy6j/_remote_module_non_scriptable.py
2023-01-11T22:20:14.5719486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9ojem59
2023-01-11T22:20:14.5720024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9ojem59/_remote_module_non_scriptable.py
2023-01-11T22:20:14.5720558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmkwwar5u
2023-01-11T22:20:14.5721077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmkwwar5u/_remote_module_non_scriptable.py
2023-01-11T22:20:14.5721598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:20:14.5722073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:20:14.5722542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:20:14.5722920Z skip: Need at least 4 CUDA devices (2.409s)
2023-01-11T22:20:14.5723285Z   test_check_overlapping (__main__.TestShardingSpec) ... ok (0.002s)
2023-01-11T22:20:14.5723690Z   test_chunked_sharding_spec (__main__.TestShardingSpec) ... ok (0.010s)
2023-01-11T22:20:14.5724070Z   test_device_placement (__main__.TestShardingSpec) ... ok (0.006s)
2023-01-11T22:20:14.5724476Z   test_enumerable_sharding_spec (__main__.TestShardingSpec) ... ok (0.007s)
2023-01-11T22:20:14.5724893Z   test_get_chunk_sharding_params (__main__.TestShardingSpec) ... ok (0.002s)
2023-01-11T22:20:14.5725465Z   test_get_chunked_dim_size (__main__.TestShardingSpec) ... ok (0.001s)
2023-01-11T22:20:14.5725840Z   test_get_split_size (__main__.TestShardingSpec) ... ok (0.001s)
2023-01-11T22:20:14.5726262Z   test_infer_sharding_spec_from_shards_metadata (__main__.TestShardingSpec) ... ok (0.009s)
2023-01-11T22:20:14.5726519Z 
2023-01-11T22:20:14.5726800Z ----------------------------------------------------------------------
2023-01-11T22:20:14.5727112Z Ran 11 tests in 8.838s
2023-01-11T22:20:14.5727274Z 
2023-01-11T22:20:14.5727383Z OK (skipped=2)
2023-01-11T22:20:14.5727536Z 
2023-01-11T22:20:14.5727659Z Generating XML reports...
2023-01-11T22:20:14.5728313Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20230111222005.xml
2023-01-11T22:20:14.5729129Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20230111222005.xml
2023-01-11T22:20:14.5729501Z 
2023-01-11T22:20:14.5729865Z ##[endgroup]
2023-01-11T22:20:14.5730546Z FINISHED PRINTING LOG FILE of distributed/_shard/sharding_spec/test_sharding_spec (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_spec-test_sharding_spec_woxh7w34)
2023-01-11T22:20:14.5730949Z 
2023-01-11T22:20:14.5731229Z Running distributed/_shard/sharded_tensor/ops/test_linear ... [2023-01-11 22:20:14.566605]
2023-01-11T22:20:14.5731982Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_linear.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:20:14.566946]
2023-01-11T22:20:25.7536935Z 
2023-01-11T22:20:25.7537764Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_linear
2023-01-11T22:20:25.7539477Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_linear (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_linear_53fvsohq)
2023-01-11T22:20:25.7539909Z 
2023-01-11T22:20:25.7540022Z Running tests...
2023-01-11T22:20:25.7540764Z ----------------------------------------------------------------------
2023-01-11T22:20:25.7541384Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear
2023-01-11T22:20:25.7541939Z   test_sharded_linear_colwise (__main__.TestShardedTensorOpsLinear) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:25.7542432Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53271
2023-01-11T22:20:25.7543116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53272
2023-01-11T22:20:25.7543812Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53273
2023-01-11T22:20:25.7544677Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53274
2023-01-11T22:20:25.7545973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:25.7546854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:25.7548121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:25.7548599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:25.7549163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:25.7549603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:25.7550172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:25.7550631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:25.7551189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:25.7551631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:25.7552890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:25.7553345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:25.7553918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:25.7554357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:25.7554915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:25.7555355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:25.7555789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:20:25.7556259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:20:25.7556735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:20:25.7557184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:20:25.7557566Z skip: Need at least 4 CUDA devices (4.109s)
2023-01-11T22:20:25.7558067Z   test_sharded_linear_errors (__main__.TestShardedTensorOpsLinear) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53407
2023-01-11T22:20:25.7558605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53408
2023-01-11T22:20:25.7559052Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53409
2023-01-11T22:20:25.7559485Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53410
2023-01-11T22:20:25.7560091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:25.7560518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:25.7561091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:25.7561555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:25.7562168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:25.7562594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:25.7563163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:25.7563623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:25.7564195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:25.7564615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:25.7565183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:25.7565639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:25.7566193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:25.7566628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:25.7567186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:25.7567644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:25.7568059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:20:25.7568554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:20:25.7569017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:20:25.7569602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:20:25.7570000Z skip: Need at least 4 CUDA devices (2.412s)
2023-01-11T22:20:25.7570502Z   test_sharded_linear_rowwise (__main__.TestShardedTensorOpsLinear) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53543
2023-01-11T22:20:25.7571053Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53544
2023-01-11T22:20:25.7571481Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53545
2023-01-11T22:20:25.7571917Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53546
2023-01-11T22:20:25.7572526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:25.7572970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:25.7573533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:25.7573999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:25.7574573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:25.7574994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:25.7575561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:25.7576018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:25.7576587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:25.7577008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:25.7577571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:25.7578030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:25.7578584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:25.7579019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:25.7579579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:25.7580036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:25.7580449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:20:25.7580922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:20:25.7581389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:20:25.7581848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:20:25.7582241Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:20:25.7582434Z 
2023-01-11T22:20:25.7582708Z ----------------------------------------------------------------------
2023-01-11T22:20:25.7583034Z Ran 3 tests in 8.932s
2023-01-11T22:20:25.7583194Z 
2023-01-11T22:20:25.7583284Z OK (skipped=3)
2023-01-11T22:20:25.7583433Z 
2023-01-11T22:20:25.7583559Z Generating XML reports...
2023-01-11T22:20:25.7584218Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20230111222016.xml
2023-01-11T22:20:25.7584616Z 
2023-01-11T22:20:25.7584939Z ##[endgroup]
2023-01-11T22:20:25.7585593Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_linear (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_linear_53fvsohq)
2023-01-11T22:20:25.7586054Z 
2023-01-11T22:20:25.7586423Z Running distributed/_shard/sharded_tensor/ops/test_elementwise_ops ... [2023-01-11 22:20:25.753781]
2023-01-11T22:20:25.7587185Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:20:25.754102]
2023-01-11T22:20:39.3874632Z 
2023-01-11T22:20:39.3875415Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_elementwise_ops
2023-01-11T22:20:39.3877465Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_elementwise_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_elementwise_ops_ype6sn72)
2023-01-11T22:20:39.3878304Z 
2023-01-11T22:20:39.3878541Z Running tests...
2023-01-11T22:20:39.3879407Z ----------------------------------------------------------------------
2023-01-11T22:20:39.3880008Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops
2023-01-11T22:20:39.3880619Z   test_sharded_dropout (__main__.TestShardedTensorElementWiseOps) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:39.3881121Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53714
2023-01-11T22:20:39.3881567Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53715
2023-01-11T22:20:39.3881986Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53716
2023-01-11T22:20:39.3882416Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53717
2023-01-11T22:20:39.3883047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:39.3883496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:39.3884054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:39.3884531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:39.3885165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:39.3885610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:39.3886148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:39.3886584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:39.3887151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:39.3887614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:39.3888175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:39.3888634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:39.3889215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:39.3889636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:39.3890203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:39.3890661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:39.3891093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:20:39.3891547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:20:39.3892006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:20:39.3892464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:20:39.3893177Z skip: Need at least 4 CUDA devices (4.097s)
2023-01-11T22:20:39.3893786Z   test_sharded_gelu (__main__.TestShardedTensorElementWiseOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53850
2023-01-11T22:20:39.3894379Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53851
2023-01-11T22:20:39.3894854Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53852
2023-01-11T22:20:39.3895311Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53853
2023-01-11T22:20:39.3895956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:39.3896434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:39.3897036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:39.3897517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:39.3898133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:39.3898603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:39.3899188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:39.3899681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:39.3900286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:39.3900752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:39.3901334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:39.3901825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:39.3902439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:39.3902914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:39.3903493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:39.3903986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:39.3904445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:20:39.3904931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:20:39.3905427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:20:39.3905923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:20:39.3906338Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:20:39.3906851Z   test_sharded_relu (__main__.TestShardedTensorElementWiseOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 53986
2023-01-11T22:20:39.3907429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 53987
2023-01-11T22:20:39.3907904Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 53988
2023-01-11T22:20:39.3908358Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 53989
2023-01-11T22:20:39.3909004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:39.3909480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:39.3910079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:39.3910556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:39.3911301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:39.3911782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:39.3912639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:39.3913136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:39.3913742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:39.3914210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:39.3914788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:39.3915261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:39.3915861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:39.3916364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:39.3916964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:39.3917456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:39.3917914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:20:39.3918397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:20:39.3918892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:20:39.3919385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:20:39.3919800Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:20:39.3920328Z   test_sharded_tensor_nan_to_num (__main__.TestShardedTensorElementWiseOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54122
2023-01-11T22:20:39.3920927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54123
2023-01-11T22:20:39.3921394Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54124
2023-01-11T22:20:39.3921850Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54125
2023-01-11T22:20:39.3922498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:39.3922976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:39.3923576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:39.3924050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:39.3924659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:39.3925137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:39.3925735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:39.3926217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:39.3926821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:39.3927286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:39.3927867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:39.3928359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:39.3928960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:39.3929605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:39.3930204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:39.3930700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:39.3931157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:20:39.3931637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:20:39.3932137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:20:39.3932629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:20:39.3933038Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:20:39.3933240Z 
2023-01-11T22:20:39.3933512Z ----------------------------------------------------------------------
2023-01-11T22:20:39.3933869Z Ran 4 tests in 11.327s
2023-01-11T22:20:39.3934037Z 
2023-01-11T22:20:39.3934146Z OK (skipped=4)
2023-01-11T22:20:39.3934305Z 
2023-01-11T22:20:39.3934415Z Generating XML reports...
2023-01-11T22:20:39.3935150Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20230111222027.xml
2023-01-11T22:20:39.3935602Z 
2023-01-11T22:20:39.3935943Z ##[endgroup]
2023-01-11T22:20:39.3936686Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_elementwise_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_elementwise_ops_ype6sn72)
2023-01-11T22:20:39.3937123Z 
2023-01-11T22:20:39.3937431Z Running distributed/tensor/parallel/test_tp_examples ... [2023-01-11 22:20:39.387539]
2023-01-11T22:20:39.3938190Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_examples.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:20:39.387790]
2023-01-11T22:20:54.6853766Z 
2023-01-11T22:20:54.6854371Z Expand the folded group to see the log file of distributed/tensor/parallel/test_tp_examples
2023-01-11T22:20:54.6855491Z ##[group]PRINTING LOG FILE of distributed/tensor/parallel/test_tp_examples (/var/lib/jenkins/workspace/test/test-reports/distributed-tensor-parallel-test_tp_examples_vti7b6xq)
2023-01-11T22:20:54.6855907Z 
2023-01-11T22:20:54.6856027Z Running tests...
2023-01-11T22:20:54.6856527Z ----------------------------------------------------------------------
2023-01-11T22:20:54.6857122Z Test results will be stored in test-reports/python-unittest/distributed.tensor.parallel.test_tp_examples
2023-01-11T22:20:54.6857686Z   test_mlp_megatron_e2e (__main__.DistTensorParallelExampleTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:20:54.6858188Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54293
2023-01-11T22:20:54.6858618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54294
2023-01-11T22:20:54.6859263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:54.6859713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:54.6860328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:54.6860782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:54.6861357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:54.6861805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:54.6862357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:54.6862824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:54.6863624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:20:54.6864126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:20:54.6864594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:20:54.6865079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:20:54.6865748Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:20:54.6866439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:20:54.6866819Z ok (5.354s)
2023-01-11T22:20:54.6867285Z   test_self_attn_megatron_e2e (__main__.DistTensorParallelExampleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54378
2023-01-11T22:20:54.6867854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54379
2023-01-11T22:20:54.6868439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:54.6868894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:54.6869473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:54.6869945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:54.6870506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:54.6870959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:54.6871535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:54.6872484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:54.6872922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:20:54.6873417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:20:54.6873905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:20:54.6874369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:20:54.6875044Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:20:54.6875739Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:20:54.6876132Z ok (3.813s)
2023-01-11T22:20:54.6876599Z   test_self_attn_replacement_megatron_e2e (__main__.DistTensorParallelExampleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54463
2023-01-11T22:20:54.6877174Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54464
2023-01-11T22:20:54.6877784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:54.6878235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:54.6878791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:54.6879259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:54.6879837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:20:54.6880262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:20:54.6880831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:20:54.6881477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:20:54.6881926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:20:54.6882400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:20:54.6882885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:20:54.6883371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:20:54.6884010Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:20:54.6884702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:20:54.6885096Z ok (3.813s)
2023-01-11T22:20:54.6885254Z 
2023-01-11T22:20:54.6885528Z ----------------------------------------------------------------------
2023-01-11T22:20:54.6885839Z Ran 3 tests in 12.981s
2023-01-11T22:20:54.6886002Z 
2023-01-11T22:20:54.6886096Z OK
2023-01-11T22:20:54.6886229Z 
2023-01-11T22:20:54.6886353Z Generating XML reports...
2023-01-11T22:20:54.6887017Z Generated XML report: test-reports/python-unittest/distributed.tensor.parallel.test_tp_examples/TEST-DistTensorParallelExampleTest-20230111222041.xml
2023-01-11T22:20:54.6887441Z 
2023-01-11T22:20:54.6887780Z ##[endgroup]
2023-01-11T22:20:54.6888446Z FINISHED PRINTING LOG FILE of distributed/tensor/parallel/test_tp_examples (/var/lib/jenkins/workspace/test/test-reports/distributed-tensor-parallel-test_tp_examples_vti7b6xq)
2023-01-11T22:20:54.6888844Z 
2023-01-11T22:20:54.6889123Z Running distributed/_shard/test_partial_tensor ... [2023-01-11 22:20:54.685412]
2023-01-11T22:20:54.6889805Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_partial_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:20:54.685670]
2023-01-11T22:21:10.4450537Z 
2023-01-11T22:21:10.4451016Z Expand the folded group to see the log file of distributed/_shard/test_partial_tensor
2023-01-11T22:21:10.4452009Z ##[group]PRINTING LOG FILE of distributed/_shard/test_partial_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_partial_tensor_so18oydm)
2023-01-11T22:21:10.4452380Z 
2023-01-11T22:21:10.4452493Z Running tests...
2023-01-11T22:21:10.4453017Z ----------------------------------------------------------------------
2023-01-11T22:21:10.4453564Z Test results will be stored in test-reports/python-unittest/distributed._shard.test_partial_tensor
2023-01-11T22:21:10.4454060Z   test_cat (__main__.TestPartialTensorOps) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:10.4454530Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54583
2023-01-11T22:21:10.4454959Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54584
2023-01-11T22:21:10.4455421Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54585
2023-01-11T22:21:10.4455865Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54586
2023-01-11T22:21:10.4456487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:10.4456979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:10.4457523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:10.4457976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:10.4458550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:10.4458998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:10.4459856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:10.4460441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:10.4461020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:10.4461452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:10.4462022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:10.4462491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:10.4463071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:10.4463498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:10.4464065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:10.4464537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:10.4464955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:21:10.4465427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:10.4465885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:21:10.4466351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:10.4466723Z skip: Need at least 4 CUDA devices (3.947s)
2023-01-11T22:21:10.4467194Z   test_cat_errors (__main__.TestPartialTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54719
2023-01-11T22:21:10.4467718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54720
2023-01-11T22:21:10.4468154Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54721
2023-01-11T22:21:10.4468598Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54722
2023-01-11T22:21:10.4469203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:10.4469651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:10.4470204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:10.4470665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:10.4471240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:10.4471680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:10.4472501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:10.4472978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:10.4473549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:10.4473971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:10.4474536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:10.4474994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:10.4475561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:10.4475980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:10.4476543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:10.4477188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:10.4477613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:21:10.4478085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:21:10.4478541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:10.4479002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:10.4479372Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:21:10.4479837Z   test_transpose (__main__.TestPartialTensorOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54855
2023-01-11T22:21:10.4480353Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54856
2023-01-11T22:21:10.4480774Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54857
2023-01-11T22:21:10.4481221Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54858
2023-01-11T22:21:10.4481828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:10.4482273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:10.4482809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:10.4483255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:10.4483814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:10.4484261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:10.4484813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:10.4485273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:10.4485862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:10.4486311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:10.4486888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:10.4487345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:10.4487918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:10.4488344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:10.4488908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:10.4489364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:10.4489783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:10.4490252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:10.4490711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:21:10.4491174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:21:10.4491545Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:21:10.4492081Z   test_partial_tensor_reshard (__main__.TestPartialTensorReshard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 54991
2023-01-11T22:21:10.4492625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 54992
2023-01-11T22:21:10.4493069Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 54993
2023-01-11T22:21:10.4493509Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 54994
2023-01-11T22:21:10.4494227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:10.4494678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:10.4495254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:10.4495705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:10.4496279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:10.4496722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:10.4497286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:10.4497730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:10.4498307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:10.4498743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:10.4499290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:10.4499750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:10.4500322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:10.4500758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:10.4501300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:10.4501760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:10.4502198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:21:10.4502668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:10.4503117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:21:10.4503570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:10.4503958Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:21:10.4504446Z   test_partial_tensor_reshard_errors (__main__.TestPartialTensorReshard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55127
2023-01-11T22:21:10.4504998Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55128
2023-01-11T22:21:10.4505442Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55129
2023-01-11T22:21:10.4505887Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55130
2023-01-11T22:21:10.4506482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:10.4506927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:10.4507497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:10.4507946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:10.4508525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:10.4508969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:10.4509537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:10.4509980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:10.4510609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:10.4511101Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:10.4511669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:10.4512384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:10.4512969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:10.4513406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:10.4513954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:10.4514417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:10.4514848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:21:10.4515324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:10.4515772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:10.4516226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:21:10.4516613Z skip: Need at least 4 CUDA devices (2.310s)
2023-01-11T22:21:10.4516806Z 
2023-01-11T22:21:10.4517061Z ----------------------------------------------------------------------
2023-01-11T22:21:10.4517390Z Ran 5 tests in 13.488s
2023-01-11T22:21:10.4517554Z 
2023-01-11T22:21:10.4517663Z OK (skipped=5)
2023-01-11T22:21:10.4517818Z 
2023-01-11T22:21:10.4517943Z Generating XML reports...
2023-01-11T22:21:10.4518541Z Generated XML report: test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorOps-20230111222056.xml
2023-01-11T22:21:10.4519354Z Generated XML report: test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorReshard-20230111222056.xml
2023-01-11T22:21:10.4519731Z 
2023-01-11T22:21:10.4520063Z ##[endgroup]
2023-01-11T22:21:10.4520657Z FINISHED PRINTING LOG FILE of distributed/_shard/test_partial_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_partial_tensor_so18oydm)
2023-01-11T22:21:10.4521017Z 
2023-01-11T22:21:10.4521321Z Running distributed/_shard/sharding_plan/test_sharding_plan ... [2023-01-11 22:21:10.445132]
2023-01-11T22:21:10.4522052Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharding_plan/test_sharding_plan.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:21:10.445390]
2023-01-11T22:21:26.2668683Z 
2023-01-11T22:21:26.2669207Z Expand the folded group to see the log file of distributed/_shard/sharding_plan/test_sharding_plan
2023-01-11T22:21:26.2670228Z ##[group]PRINTING LOG FILE of distributed/_shard/sharding_plan/test_sharding_plan (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_plan-test_sharding_plan_nhy9zj6d)
2023-01-11T22:21:26.2670651Z 
2023-01-11T22:21:26.2670768Z Running tests...
2023-01-11T22:21:26.2671273Z ----------------------------------------------------------------------
2023-01-11T22:21:26.2672150Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan
2023-01-11T22:21:26.2672741Z   test_custom_sharding_planner (__main__.TestShardingPlan) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:26.2673220Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55298
2023-01-11T22:21:26.2673667Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55299
2023-01-11T22:21:26.2674099Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55300
2023-01-11T22:21:26.2674543Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55301
2023-01-11T22:21:26.2675435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:26.2675992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:26.2676563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:26.2677041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:26.2677631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:26.2678065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:26.2678724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:26.2679173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:26.2679755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:26.2680209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:26.2680777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:26.2681222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:26.2681793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:26.2682234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:26.2682807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:26.2683248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:26.2683678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:21:26.2684153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:26.2684605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:21:26.2685062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:26.2685446Z skip: Need at least 4 CUDA devices (4.103s)
2023-01-11T22:21:26.2685931Z   test_reshard_to_ddp_sharding_plan (__main__.TestShardingPlan) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55434
2023-01-11T22:21:26.2686442Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55435
2023-01-11T22:21:26.2686879Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55436
2023-01-11T22:21:26.2687313Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55437
2023-01-11T22:21:26.2687900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:26.2688351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:26.2688925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:26.2689387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:26.2689943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:26.2690386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:26.2690941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:26.2691367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:26.2691933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:26.2692519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:26.2693111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:26.2693553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:26.2694119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:26.2694559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:26.2695125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:26.2695568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:26.2695998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:26.2696467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:26.2696919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:21:26.2697381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:21:26.2697759Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:21:26.2698240Z   test_shard_module_sub_process_group (__main__.TestShardingPlan) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55570
2023-01-11T22:21:26.2698752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55571
2023-01-11T22:21:26.2699192Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55572
2023-01-11T22:21:26.2699631Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55573
2023-01-11T22:21:26.2700217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:26.2700669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:26.2701234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:26.2701696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:26.2702249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:26.2702688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:26.2703257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:26.2703716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:26.2704275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:26.2704710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:26.2705279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:26.2705720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:26.2706292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:26.2706728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:26.2707286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:26.2707722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:26.2708150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:26.2708621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:26.2709188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:21:26.2709658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:21:26.2710052Z skip: Need at least 4 CUDA devices (2.310s)
2023-01-11T22:21:26.2710523Z   test_sharding_plan_errors (__main__.TestShardingPlan) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55706
2023-01-11T22:21:26.2711026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55707
2023-01-11T22:21:26.2711472Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55708
2023-01-11T22:21:26.2712223Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55709
2023-01-11T22:21:26.2712838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:26.2713287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:26.2713852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:26.2714291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:26.2714842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:26.2715303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:26.2715883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:26.2716344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:26.2716897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:26.2717339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:26.2717908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:26.2718349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:26.2718916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:26.2719355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:26.2719919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:26.2720358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:26.2720788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:26.2721254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:21:26.2721697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:21:26.2722165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:26.2722546Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:21:26.2723022Z   test_sharding_plan_simple_megatron (__main__.TestShardingPlan) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55842
2023-01-11T22:21:26.2723533Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55843
2023-01-11T22:21:26.2723975Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55844
2023-01-11T22:21:26.2724409Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55845
2023-01-11T22:21:26.2725015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:26.2725442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:26.2726100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:26.2726630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:26.2727188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:26.2727629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:26.2728195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:26.2728658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:26.2729207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:26.2729647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:26.2730207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:26.2730655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:26.2731221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:26.2731657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:26.2732215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:26.2732653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:26.2733090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:21:26.2733558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:26.2734025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:26.2734477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:21:26.2734861Z skip: Need at least 4 CUDA devices (2.312s)
2023-01-11T22:21:26.2735051Z 
2023-01-11T22:21:26.2735327Z ----------------------------------------------------------------------
2023-01-11T22:21:26.2735638Z Ran 5 tests in 13.546s
2023-01-11T22:21:26.2735797Z 
2023-01-11T22:21:26.2735906Z OK (skipped=5)
2023-01-11T22:21:26.2736058Z 
2023-01-11T22:21:26.2736180Z Generating XML reports...
2023-01-11T22:21:26.2736800Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan/TEST-TestShardingPlan-20230111222112.xml
2023-01-11T22:21:26.2737147Z 
2023-01-11T22:21:26.2737472Z ##[endgroup]
2023-01-11T22:21:26.2738137Z FINISHED PRINTING LOG FILE of distributed/_shard/sharding_plan/test_sharding_plan (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_plan-test_sharding_plan_nhy9zj6d)
2023-01-11T22:21:26.2738542Z 
2023-01-11T22:21:26.2738831Z Running distributed/_composable/test_replicate ... [2023-01-11 22:21:26.266929]
2023-01-11T22:21:26.2739510Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_replicate.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:21:26.267175]
2023-01-11T22:21:42.7474833Z 
2023-01-11T22:21:42.7475649Z Expand the folded group to see the log file of distributed/_composable/test_replicate
2023-01-11T22:21:42.7476599Z ##[group]PRINTING LOG FILE of distributed/_composable/test_replicate (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_replicate_460e8tpj)
2023-01-11T22:21:42.7476972Z 
2023-01-11T22:21:42.7477086Z Running tests...
2023-01-11T22:21:42.7477621Z ----------------------------------------------------------------------
2023-01-11T22:21:42.7478288Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_replicate
2023-01-11T22:21:42.7479094Z   test_replicate_non_root_multiple_save_load (__main__.ReplicateStateDictTest)
2023-01-11T22:21:42.7479962Z Tests tha replicate() on multiple submodules matches ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:42.7480445Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56013
2023-01-11T22:21:42.7480876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56014
2023-01-11T22:21:42.7481712Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56015
2023-01-11T22:21:42.7482225Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56016
2023-01-11T22:21:42.7482858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:42.7483292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:42.7483865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:42.7484332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:42.7484901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:42.7485346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:42.7485917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:42.7486378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:42.7487035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:42.7487482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:42.7488032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:42.7488500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:42.7489082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:42.7489504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:42.7490071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:42.7490532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:42.7490967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:42.7491417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:21:42.7491870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:21:42.7492331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:42.7492655Z ok (4.099s)
2023-01-11T22:21:42.7493001Z   test_replicate_single_module_save_load (__main__.ReplicateStateDictTest)
2023-01-11T22:21:42.7493521Z Tests that replicate() on a single module state_dict ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56149
2023-01-11T22:21:42.7494037Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56150
2023-01-11T22:21:42.7494459Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56151
2023-01-11T22:21:42.7494893Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56152
2023-01-11T22:21:42.7495503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:42.7495932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:42.7496505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:42.7496969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:42.7497689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:42.7498122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:42.7498686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:42.7499149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:42.7500005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:42.7500432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:42.7501008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:42.7501466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:42.7502029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:42.7502468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:42.7503035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:42.7503491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:42.7503905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:42.7504374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:21:42.7504840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:21:42.7505286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:42.7505629Z ok (2.410s)
2023-01-11T22:21:42.7506065Z   test_replicate_multi_module (__main__.ReplicateTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56285
2023-01-11T22:21:42.7506582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56286
2023-01-11T22:21:42.7507003Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56287
2023-01-11T22:21:42.7507435Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56288
2023-01-11T22:21:42.7508040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:42.7508466Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:42.7509034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:42.7509497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:42.7510079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:42.7510504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:42.7511072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:42.7511531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:42.7512304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:42.7512725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:42.7513300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:42.7513757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:42.7514309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:42.7514925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:42.7515499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:42.7515956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:42.7516366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:21:42.7516837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:42.7517298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:42.7517744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:21:42.7518231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:42.7518728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:42.7519222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:21:42.7519684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:21:42.7520333Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:21:42.7521019Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:21:42.7521699Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:21:42.7522354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:21:42.7522880Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:42.7523358Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:42.7523828Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:42.7524270Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:42.7524739Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:42.7525204Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:42.7525645Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:42.7526107Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:42.7526573Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:42.7527042Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:42.7527479Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:42.7527942Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:42.7528282Z ok (2.510s)
2023-01-11T22:21:42.7528693Z   test_replicate_single_module (__main__.ReplicateTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56441
2023-01-11T22:21:42.7529215Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56442
2023-01-11T22:21:42.7529656Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56443
2023-01-11T22:21:42.7530091Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56444
2023-01-11T22:21:42.7530683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:42.7531244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:42.7531821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:42.7532271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:42.7532847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:42.7533287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:42.7533852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:42.7534296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:42.7534874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:42.7535323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:42.7535890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:42.7536331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:42.7536899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:42.7537337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:42.7537883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:42.7538343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:42.7538773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:42.7539242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:21:42.7539695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:42.7540152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:21:42.7540631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:42.7541103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:42.7541589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:21:42.7542070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:21:42.7542718Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:21:42.7543388Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:21:42.7544070Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:21:42.7544748Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:21:42.7545270Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:42.7545725Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:42.7546192Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:42.7546703Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:42.7547040Z ok (2.509s)
2023-01-11T22:21:42.7547503Z   test_replicate_with_kwargs (__main__.ReplicateTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56597
2023-01-11T22:21:42.7548070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56598
2023-01-11T22:21:42.7548512Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56599
2023-01-11T22:21:42.7548928Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56600
2023-01-11T22:21:42.7549534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:42.7549978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:42.7550546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:42.7550994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:42.7551566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:42.7552322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:42.7552887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:42.7553349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:42.7553921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:42.7554359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:42.7554902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:42.7555366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:42.7555937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:42.7556376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:42.7556926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:42.7557385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:42.7557818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:42.7558267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:21:42.7558723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:21:42.7559186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:42.7559666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:42.7560141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:42.7560633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:21:42.7561120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:21:42.7561750Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:21:42.7562435Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:21:42.7563113Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:21:42.7563790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:21:42.7564285Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:42.7564915Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:42.7565394Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:42.7565862Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:42.7566181Z ok (2.610s)
2023-01-11T22:21:42.7566331Z 
2023-01-11T22:21:42.7566606Z ----------------------------------------------------------------------
2023-01-11T22:21:42.7566935Z Ran 5 tests in 14.138s
2023-01-11T22:21:42.7567096Z 
2023-01-11T22:21:42.7567170Z OK
2023-01-11T22:21:42.7567303Z 
2023-01-11T22:21:42.7567426Z Generating XML reports...
2023-01-11T22:21:42.7568050Z Generated XML report: test-reports/python-unittest/distributed._composable.test_replicate/TEST-ReplicateStateDictTest-20230111222128.xml
2023-01-11T22:21:42.7568820Z Generated XML report: test-reports/python-unittest/distributed._composable.test_replicate/TEST-ReplicateTest-20230111222128.xml
2023-01-11T22:21:42.7569166Z 
2023-01-11T22:21:42.7569484Z ##[endgroup]
2023-01-11T22:21:42.7570100Z FINISHED PRINTING LOG FILE of distributed/_composable/test_replicate (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_replicate_460e8tpj)
2023-01-11T22:21:42.7570466Z 
2023-01-11T22:21:42.7570757Z Running distributed/fsdp/test_fsdp_tp_integration ... [2023-01-11 22:21:42.747607]
2023-01-11T22:21:42.7571443Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_tp_integration.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:21:42.747866]
2023-01-11T22:22:03.2182290Z 
2023-01-11T22:22:03.2182817Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_tp_integration
2023-01-11T22:22:03.2185865Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_tp_integration (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_tp_integration_g9ba1q89)
2023-01-11T22:22:03.2186302Z 
2023-01-11T22:22:03.2188767Z Running tests...
2023-01-11T22:22:03.2189967Z ----------------------------------------------------------------------
2023-01-11T22:22:03.2190833Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration
2023-01-11T22:22:03.2191329Z   test_fsdp_tp_checkpoint_integration (__main__.TestTPFSDPIntegration)
2023-01-11T22:22:03.2191822Z Tests checkpointing for TP + FSDP integration. ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:22:03.2192689Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56788
2023-01-11T22:22:03.2193507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56789
2023-01-11T22:22:03.2194167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:03.2194630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:03.2195196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:03.2195682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:03.2196262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:03.2196723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:03.2197283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:03.2197752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:03.2198207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:22:03.2198702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:22:03.2200088Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:22:03.2201175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:22:03.2201716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:22:03.2202186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:22:03.2202551Z dist init r=1, world=2
2023-01-11T22:22:03.2202784Z dist init r=0, world=2
2023-01-11T22:22:03.2203069Z skip: Need at least 4 CUDA devices (4.993s)
2023-01-11T22:22:03.2203535Z   test_fsdp_tp_integration_tensor_parallel_size_2_cpu_offload_CPUOffload(offload_params=False) (__main__.TestTPFSDPIntegration)
2023-01-11T22:22:03.2204273Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56867
2023-01-11T22:22:03.2204815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56868
2023-01-11T22:22:03.2205434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:03.2205889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:03.2206438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:03.2206914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:03.2207496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:03.2207941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:03.2208489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:03.2208946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:03.2209405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:22:03.2209883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:22:03.2210539Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:22:03.2211224Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:22:03.2211745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:22:03.2212196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:22:03.2212539Z dist init r=0, world=2
2023-01-11T22:22:03.2212829Z dist init r=1, world=2
2023-01-11T22:22:03.2213114Z skip: Need at least 4 CUDA devices (3.210s)
2023-01-11T22:22:03.2213557Z   test_fsdp_tp_integration_tensor_parallel_size_2_cpu_offload_CPUOffload(offload_params=True) (__main__.TestTPFSDPIntegration)
2023-01-11T22:22:03.2214306Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56946
2023-01-11T22:22:03.2214836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56947
2023-01-11T22:22:03.2215440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:03.2215869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:03.2216438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:03.2216900Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:03.2217466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:03.2218023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:03.2218594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:03.2219052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:03.2219484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:22:03.2219975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:22:03.2220624Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:22:03.2221307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:22:03.2221806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:22:03.2222280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:22:03.2222631Z dist init r=0, world=2
2023-01-11T22:22:03.2222863Z dist init r=1, world=2
2023-01-11T22:22:03.2223146Z skip: Need at least 4 CUDA devices (3.312s)
2023-01-11T22:22:03.2223599Z   test_fsdp_tp_integration_tensor_parallel_size_4_cpu_offload_CPUOffload(offload_params=False) (__main__.TestTPFSDPIntegration)
2023-01-11T22:22:03.2224338Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57025
2023-01-11T22:22:03.2224850Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57026
2023-01-11T22:22:03.2225453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:03.2225896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:03.2226457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:03.2226924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:03.2227500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:03.2227936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:03.2228483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:03.2228946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:03.2229384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:22:03.2229872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:22:03.2230526Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:22:03.2231209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:22:03.2231708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:22:03.2232593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:22:03.2232952Z dist init r=1, world=2
2023-01-11T22:22:03.2233206Z dist init r=0, world=2
2023-01-11T22:22:03.2233473Z skip: Need at least 4 CUDA devices (3.311s)
2023-01-11T22:22:03.2233927Z   test_fsdp_tp_integration_tensor_parallel_size_4_cpu_offload_CPUOffload(offload_params=True) (__main__.TestTPFSDPIntegration)
2023-01-11T22:22:03.2234675Z Tests training for TP + FSDP integration by comparing an FSDP-only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57104
2023-01-11T22:22:03.2235353Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57105
2023-01-11T22:22:03.2235967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:03.2236414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:03.2236982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:03.2237428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:03.2238003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:03.2238444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:03.2238990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:03.2239453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:03.2239910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:22:03.2240402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:22:03.2241030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:22:03.2241712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:22:03.2242226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:22:03.2242687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:22:03.2243020Z dist init r=1, world=2
2023-01-11T22:22:03.2243269Z dist init r=0, world=2
2023-01-11T22:22:03.2243551Z skip: Need at least 4 CUDA devices (3.312s)
2023-01-11T22:22:03.2243730Z 
2023-01-11T22:22:03.2244007Z ----------------------------------------------------------------------
2023-01-11T22:22:03.2244333Z Ran 5 tests in 18.138s
2023-01-11T22:22:03.2244490Z 
2023-01-11T22:22:03.2244601Z OK (skipped=5)
2023-01-11T22:22:03.2244752Z 
2023-01-11T22:22:03.2244858Z Generating XML reports...
2023-01-11T22:22:03.2245484Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration/TEST-TestTPFSDPIntegration-20230111222144.xml
2023-01-11T22:22:03.2245854Z 
2023-01-11T22:22:03.2246269Z ##[endgroup]
2023-01-11T22:22:03.2246903Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_tp_integration (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_tp_integration_g9ba1q89)
2023-01-11T22:22:03.2247275Z 
2023-01-11T22:22:03.2247526Z Running distributed/test_dynamo_distributed ... [2023-01-11 22:22:03.218317]
2023-01-11T22:22:03.2248270Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_dynamo_distributed.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:22:03.218668]
2023-01-11T22:22:27.2208964Z 
2023-01-11T22:22:27.2209814Z Expand the folded group to see the log file of distributed/test_dynamo_distributed
2023-01-11T22:22:27.2211999Z ##[group]PRINTING LOG FILE of distributed/test_dynamo_distributed (/var/lib/jenkins/workspace/test/test-reports/distributed-test_dynamo_distributed_i0o5gx5v)
2023-01-11T22:22:27.2212393Z 
2023-01-11T22:22:27.2212505Z Running tests...
2023-01-11T22:22:27.2215051Z ----------------------------------------------------------------------
2023-01-11T22:22:27.2215557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:22:27.2216488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:22:27.2217157Z Test results will be stored in test-reports/python-unittest/distributed.test_dynamo_distributed
2023-01-11T22:22:27.2217976Z   test_aot_autograd (__main__.TestDistributed)
2023-01-11T22:22:27.2218521Z Explicitly check AotAutograd family of compilers work, ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:22:27.2219322Z [2023-01-11 22:22:08,126] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored
2023-01-11T22:22:27.2219671Z ok (3.063s)
2023-01-11T22:22:27.2220084Z   test_custom_layer (__main__.TestDistributed)
2023-01-11T22:22:27.2220812Z Just ensures that the appropriate number of splits happen (based on ... [2023-01-11 22:22:08,613] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored
2023-01-11T22:22:27.2221480Z frames [('total', 4), ('ok', 4)]
2023-01-11T22:22:27.2221973Z inline_call [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)]
2023-01-11T22:22:27.2222311Z unimplemented []
2023-01-11T22:22:27.2222982Z graph_break [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)]
2023-01-11T22:22:27.2224306Z stats [('calls_captured', 8), ('fusions_possible', 7), ('unique_graphs', 1)]
2023-01-11T22:22:27.2224727Z aot_autograd [('total', 3), ('ok', 3)]
2023-01-11T22:22:27.2225079Z frames [('total', 2), ('ok', 2)]
2023-01-11T22:22:27.2225752Z inline_call [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)]
2023-01-11T22:22:27.2226106Z unimplemented []
2023-01-11T22:22:27.2226699Z graph_break [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)]
2023-01-11T22:22:27.2227270Z stats [('calls_captured', 7), ('fusions_possible', 6), ('unique_graphs', 1)]
2023-01-11T22:22:27.2227581Z ok (0.086s)
2023-01-11T22:22:27.2228228Z   test_ddp_baseline_aot_eager (__main__.TestDistributed) ... frames [('total', 1), ('ok', 1)]
2023-01-11T22:22:27.2228730Z stats [('calls_captured', 8), ('fusions_possible', 7), ('unique_graphs', 1)]
2023-01-11T22:22:27.2229299Z aot_autograd [('total', 1), ('ok', 1)]
2023-01-11T22:22:27.2229606Z ok (0.775s)
2023-01-11T22:22:27.2229992Z   test_ddp_baseline_inductor (__main__.TestDistributed) ... skip: Inductor+gpu needs triton and recent GPU arch (0.001s)
2023-01-11T22:22:27.2230757Z   test_empty_graph_inductor (__main__.TestDistributed) ... skip: Inductor+gpu needs triton and recent GPU arch (0.000s)
2023-01-11T22:22:27.2231326Z   test_fsdp_orig_params_assert (__main__.TestDistributed) ... frames [('total', 1)]
2023-01-11T22:22:27.2231673Z ok (1.467s)
2023-01-11T22:22:27.2232438Z   test_graph_split (__main__.TestDistributed)
2023-01-11T22:22:27.2233484Z Just ensures that the appropriate number of splits happen (based on ... [2023-01-11 22:22:11,463] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored
2023-01-11T22:22:27.2233975Z frames [('total', 2), ('ok', 2)]
2023-01-11T22:22:27.2234654Z inline_call [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)]
2023-01-11T22:22:27.2235024Z unimplemented []
2023-01-11T22:22:27.2235503Z graph_break [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)]
2023-01-11T22:22:27.2236232Z stats [('calls_captured', 8), ('fusions_possible', 7), ('unique_graphs', 1)]
2023-01-11T22:22:27.2236555Z ok (0.634s)
2023-01-11T22:22:27.2236871Z   test_graph_split_inductor (__main__.TestDistributed)
2023-01-11T22:22:27.2237500Z Same as above, but using inductor backend. ... skip: Inductor+gpu needs triton and recent GPU arch (0.001s)
2023-01-11T22:22:27.2237919Z   test_ignored_parameters (__main__.TestDistributed)
2023-01-11T22:22:27.2238817Z Verifies ddp graph-split logic ignores parameters marked to ignore on DDP module. ... [2023-01-11 22:22:11,578] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored
2023-01-11T22:22:27.2239326Z frames [('total', 2), ('ok', 2)]
2023-01-11T22:22:27.2239980Z inline_call [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)]
2023-01-11T22:22:27.2240339Z unimplemented []
2023-01-11T22:22:27.2240831Z graph_break [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)]
2023-01-11T22:22:27.2241498Z stats [('calls_captured', 7), ('fusions_possible', 6), ('unique_graphs', 1)]
2023-01-11T22:22:27.2241957Z ok (0.082s)
2023-01-11T22:22:27.2242522Z   test_no_split (__main__.TestDistributed)
2023-01-11T22:22:27.2243185Z Ensures the DDPOptimizer returns a correct, compiled module without ... [2023-01-11 22:22:11,650] torch._dynamo.variables.torch: [WARNING] Profiler will be ignored
2023-01-11T22:22:27.2243879Z frames [('total', 2), ('ok', 2)]
2023-01-11T22:22:27.2244329Z inline_call [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)]
2023-01-11T22:22:27.2244702Z unimplemented []
2023-01-11T22:22:27.2245320Z graph_break [('call_function UserDefinedObjectVariable(instancemethod) [] {}', 1)]
2023-01-11T22:22:27.2245825Z stats [('calls_captured', 8), ('fusions_possible', 7), ('unique_graphs', 1)]
2023-01-11T22:22:27.2246317Z ok (0.057s)
2023-01-11T22:22:27.2246798Z   test_ddp_baseline_aot_eager_multiprocess (__main__.TestDistributedMultiProc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57299
2023-01-11T22:22:27.2247480Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57300
2023-01-11T22:22:27.2248210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:27.2248797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:27.2249444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:27.2250075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:27.2250758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:27.2251268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:27.2251983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:27.2252457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:27.2253113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:22:27.2253745Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2023-01-11T22:22:27.2254457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:22:27.2254940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:22:27.2255640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:22:27.2256278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:22:27.2257193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:22:27.2257583Z ok (6.117s)
2023-01-11T22:22:27.2258242Z   test_fsdp_aot_eager (__main__.TestDistributedMultiProc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57389
2023-01-11T22:22:27.2258762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57390
2023-01-11T22:22:27.2259596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:27.2260044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:27.2260817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:27.2261291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:27.2262086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:27.2262537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:27.2263471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:27.2263952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:27.2264620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:22:27.2265080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:22:27.2265710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:22:27.2266324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:22:27.2267171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:22:27.2267874Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:22:27.2268443Z ok (5.915s)
2023-01-11T22:22:27.2268897Z   test_fsdp_inductor (__main__.TestDistributedMultiProc) ... skip: Inductor+gpu needs triton and recent GPU arch (0.001s)
2023-01-11T22:22:27.2269578Z   test_hf_bert_ddp_aot_eager (__main__.TestDistributedMultiProc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57479
2023-01-11T22:22:27.2270224Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57480
2023-01-11T22:22:27.2270973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:27.2271468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:27.2272618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:27.2273078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:27.2273678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:27.2274343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:27.2274922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:27.2275359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:27.2275789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:22:27.2276445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:22:27.2276865Z skip: Test skipped due to missing import (2.810s)
2023-01-11T22:22:27.2277331Z   test_hf_bert_ddp_inductor (__main__.TestDistributedMultiProc) ... skip: Inductor+gpu needs triton and recent GPU arch (0.001s)
2023-01-11T22:22:27.2277882Z   test_hf_bert_fsdp (__main__.TestDistributedMultiProc) ... skip: Inductor+gpu needs triton and recent GPU arch (0.001s)
2023-01-11T22:22:27.2278441Z   test_hf_bert_ddp_aot_eager (__main__.TestFakeDistributedSingleProc) ... skip: Unable to import transformers (0.001s)
2023-01-11T22:22:27.2278997Z   test_hf_bert_ddp_inductor (__main__.TestFakeDistributedSingleProc) ... skip: Inductor+gpu needs triton and recent GPU arch (0.000s)
2023-01-11T22:22:27.2279628Z   test_issue90375 (__main__.TestFakeDistributedSingleProc) ... frames [('total', 4), ('ok', 3)]
2023-01-11T22:22:27.2280260Z inline_call [('inline in skipfiles: helper /opt/conda/lib/python3.10/contextlib.py', 1)]
2023-01-11T22:22:27.2281020Z unimplemented [('SETUP_WITH UserDefinedObjectVariable(_GeneratorContextManager)', 1), ('generator', 1)]
2023-01-11T22:22:27.2281584Z graph_break [('inline in skipfiles: helper /opt/conda/lib/python3.10/contextlib.py', 1)]
2023-01-11T22:22:27.2282083Z stats [('calls_captured', 3), ('fusions_possible', 2), ('unique_graphs', 1)]
2023-01-11T22:22:27.2282468Z aot_autograd [('total', 1), ('ok', 1)]
2023-01-11T22:22:27.2282835Z ok (0.064s)
2023-01-11T22:22:27.2282980Z 
2023-01-11T22:22:27.2283310Z ----------------------------------------------------------------------
2023-01-11T22:22:27.2283648Z Ran 19 tests in 21.082s
2023-01-11T22:22:27.2283810Z 
2023-01-11T22:22:27.2283918Z OK (skipped=9)
2023-01-11T22:22:27.2284053Z 
2023-01-11T22:22:27.2284175Z Generating XML reports...
2023-01-11T22:22:27.2284760Z Generated XML report: test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestDistributed-20230111222205.xml
2023-01-11T22:22:27.2285532Z Generated XML report: test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestDistributedMultiProc-20230111222205.xml
2023-01-11T22:22:27.2286344Z Generated XML report: test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestFakeDistributedSingleProc-20230111222205.xml
2023-01-11T22:22:27.2286730Z 
2023-01-11T22:22:27.2287171Z ##[endgroup]
2023-01-11T22:22:27.2288054Z FINISHED PRINTING LOG FILE of distributed/test_dynamo_distributed (/var/lib/jenkins/workspace/test/test-reports/distributed-test_dynamo_distributed_i0o5gx5v)
2023-01-11T22:22:27.2288419Z 
2023-01-11T22:22:27.2288703Z Running distributed/checkpoint/test_checkpoint ... [2023-01-11 22:22:27.221014]
2023-01-11T22:22:27.2289381Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:22:27.221289]
2023-01-11T22:22:52.0576181Z 
2023-01-11T22:22:52.0579115Z Expand the folded group to see the log file of distributed/checkpoint/test_checkpoint
2023-01-11T22:22:52.0580101Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_checkpoint_mz8vwl3j)
2023-01-11T22:22:52.0582473Z 
2023-01-11T22:22:52.0582875Z Running tests...
2023-01-11T22:22:52.0583814Z ----------------------------------------------------------------------
2023-01-11T22:22:52.0584576Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_checkpoint
2023-01-11T22:22:52.0585165Z   test_default_metadata (__main__.TestDistributedCheckpointing) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:22:52.0585658Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57585
2023-01-11T22:22:52.0586113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57586
2023-01-11T22:22:52.0586740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0587199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0587769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0590530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0591206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0591704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0592747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0593243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0593671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:22:52.0594174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:22:52.0594656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:22:52.0595128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:22:52.0595791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:22:52.0596836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:22:52.0597235Z ok (4.833s)
2023-01-11T22:22:52.0597701Z   test_tensor_metadata_with_missing_rank_spec (__main__.TestDistributedCheckpointing) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57664
2023-01-11T22:22:52.0598282Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57665
2023-01-11T22:22:52.0598896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0599350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0600004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0600483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0601046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0601498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0602069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0602518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0602954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:22:52.0603425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:22:52.0603912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:22:52.0604387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:22:52.0605048Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:22:52.0605739Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:22:52.0606109Z ok (3.209s)
2023-01-11T22:22:52.0606549Z   test_dummy_reader_works (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57743
2023-01-11T22:22:52.0607080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57744
2023-01-11T22:22:52.0607521Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 57745
2023-01-11T22:22:52.0607949Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 57746
2023-01-11T22:22:52.0608552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0609000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0609576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0610029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0610604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0611045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0611593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0612056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0612631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0613072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0613685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0614196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0614772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0615195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0615760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0616220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0616652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:22:52.0617103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:22:52.0617560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:22:52.0618030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:22:52.0618424Z skip: Need at least 4 CUDA devices (2.409s)
2023-01-11T22:22:52.0618883Z   test_dummy_writer_works (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57879
2023-01-11T22:22:52.0619416Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57880
2023-01-11T22:22:52.0619854Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 57881
2023-01-11T22:22:52.0620272Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 57882
2023-01-11T22:22:52.0620877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0621324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0621894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0622350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0622923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0623366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0623913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0624377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0624953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0625390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0625938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0626413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0626987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0627426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0627968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0628431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0628859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:22:52.0629309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:22:52.0629768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:22:52.0630293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:22:52.0630731Z skip: Need at least 4 CUDA devices (2.409s)
2023-01-11T22:22:52.0631194Z   test_load_error_handling (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58015
2023-01-11T22:22:52.0631724Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58016
2023-01-11T22:22:52.0632461Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 58017
2023-01-11T22:22:52.0632888Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 58018
2023-01-11T22:22:52.0633496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0633944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0634518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0634973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0635547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0635989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0636541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0637005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0637574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0638012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0638559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0639024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0639596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0640035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0640576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0641035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0641465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:22:52.0641910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:22:52.0642379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:22:52.0642836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:22:52.0643234Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:22:52.0643707Z   test_load_error_handling_no_dist (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58151
2023-01-11T22:22:52.0644245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58152
2023-01-11T22:22:52.0644689Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 58153
2023-01-11T22:22:52.0645111Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 58154
2023-01-11T22:22:52.0645720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0646170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0646737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0647183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0647920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0648371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0648940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0649386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0649959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0650399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0650943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0651446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0652030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0652474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0653020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0653480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0653914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:22:52.0654365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:22:52.0654827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:22:52.0655281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:22:52.0655624Z ok (2.409s)
2023-01-11T22:22:52.0656053Z   test_save_error_handling (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58287
2023-01-11T22:22:52.0656582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58288
2023-01-11T22:22:52.0657027Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 58289
2023-01-11T22:22:52.0657446Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 58290
2023-01-11T22:22:52.0658052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0658499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0659068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0659514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0660101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0660543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0661111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0661554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0662132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0662571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0663116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0663576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0664143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0664722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0665277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0665739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0666176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:22:52.0666649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:22:52.0667094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:22:52.0667552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:22:52.0667948Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:22:52.0668422Z   test_save_error_handling_no_dist (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58423
2023-01-11T22:22:52.0668970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58424
2023-01-11T22:22:52.0669417Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 58425
2023-01-11T22:22:52.0669858Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 58426
2023-01-11T22:22:52.0670442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0670890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0671458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0672302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0672884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0673338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0673908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0674353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0674930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0675370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0675936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0676375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0676947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:52.0677392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:52.0677963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:52.0678403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:52.0678839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:22:52.0679307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:22:52.0679755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:22:52.0680216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:22:52.0680563Z ok (2.409s)
2023-01-11T22:22:52.0680715Z 
2023-01-11T22:22:52.0680992Z ----------------------------------------------------------------------
2023-01-11T22:22:52.0681409Z Ran 8 tests in 22.501s
2023-01-11T22:22:52.0681572Z 
2023-01-11T22:22:52.0681739Z OK (skipped=4)
2023-01-11T22:22:52.0681897Z 
2023-01-11T22:22:52.0682024Z Generating XML reports...
2023-01-11T22:22:52.0682670Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedCheckpointing-20230111222229.xml
2023-01-11T22:22:52.0683514Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedFailure-20230111222229.xml
2023-01-11T22:22:52.0683899Z 
2023-01-11T22:22:52.0684353Z ##[endgroup]
2023-01-11T22:22:52.0684988Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_checkpoint_mz8vwl3j)
2023-01-11T22:22:52.0685347Z 
2023-01-11T22:22:52.0685645Z Running distributed/elastic/multiprocessing/api_test ... [2023-01-11 22:22:52.057778]
2023-01-11T22:22:52.0686370Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/multiprocessing/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:22:52.058053]
2023-01-11T22:23:21.7221834Z 
2023-01-11T22:23:21.7224455Z Expand the folded group to see the log file of distributed/elastic/multiprocessing/api_test
2023-01-11T22:23:21.7225490Z ##[group]PRINTING LOG FILE of distributed/elastic/multiprocessing/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-multiprocessing-api_test_p63dzfto)
2023-01-11T22:23:21.7225912Z 
2023-01-11T22:23:21.7226027Z Running tests...
2023-01-11T22:23:21.7226519Z ----------------------------------------------------------------------
2023-01-11T22:23:21.7228038Z Test results will be stored in test-reports/python-unittest/distributed.elastic.multiprocessing.api_test
2023-01-11T22:23:21.7228532Z   test_get_failures (__main__.RunProcResultsTest) ... ok (1.670s)
2023-01-11T22:23:21.7228909Z   test_is_failed (__main__.RunProcResultsTest) ... ok (0.001s)
2023-01-11T22:23:21.7229291Z   test_args_env_len_mismatch (__main__.StartProcessesListTest) ... ok (0.002s)
2023-01-11T22:23:21.7229744Z   test_binary (__main__.StartProcessesListTest) ... hello stderr from 0
2023-01-11T22:23:21.7231983Z hello stdout from 0
2023-01-11T22:23:21.7232727Z hello stderr from 1
2023-01-11T22:23:21.7232996Z hello stdout from 1
2023-01-11T22:23:21.7233239Z ok (0.104s)
2023-01-11T22:23:21.7233560Z   test_binary_exit (__main__.StartProcessesListTest) ... bar stderr from 1
2023-01-11T22:23:21.7233895Z bar stdout from 1
2023-01-11T22:23:21.7242286Z failed (exitcode: 138) local_rank: 0 (pid: 58595) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo1.py
2023-01-11T22:23:21.7242696Z ok (0.104s)
2023-01-11T22:23:21.7243069Z   test_binary_incorrect_entrypoint (__main__.StartProcessesListTest) ... ok (0.002s)
2023-01-11T22:23:21.7243519Z   test_binary_raises (__main__.StartProcessesListTest) ... bar from 1
2023-01-11T22:23:21.7243851Z Traceback (most recent call last):
2023-01-11T22:23:21.7244290Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py", line 22, in <module>
2023-01-11T22:23:21.7244715Z     raise RuntimeError(f"raised from {rank}")
2023-01-11T22:23:21.7245020Z RuntimeError: raised from 0
2023-01-11T22:23:21.7245448Z failed (exitcode: 1) local_rank: 0 (pid: 58598) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py
2023-01-11T22:23:21.7245846Z ok (0.104s)
2023-01-11T22:23:21.7246287Z   test_binary_redirect_and_tee (__main__.StartProcessesListTest) ... world stdout from 1
2023-01-11T22:23:21.7246662Z [trainer1]:world stderr from 1
2023-01-11T22:23:21.7246940Z [trainer0]:hello stdout from 0
2023-01-11T22:23:21.7247180Z ok (1.106s)
2023-01-11T22:23:21.7247858Z   test_function (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:21.7248435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:21.7249329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:21.7249815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:21.7250397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:21.7250842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:21.7251397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:21.7251861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:21.7252176Z hello stderr from 0
2023-01-11T22:23:21.7252405Z hello stderr from 1
2023-01-11T22:23:21.7252647Z hello stdout from 1
2023-01-11T22:23:21.7252888Z hello stdout from 0
2023-01-11T22:23:21.7253142Z Closing process 58606 via signal SIGTERM
2023-01-11T22:23:21.7253411Z ok (2.419s)
2023-01-11T22:23:21.7254100Z   test_function_large_ret_val (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:21.7254637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:21.7255193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:21.7255657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:21.7256226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:21.7256652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:21.7257218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:21.7257680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:21.7258259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:21.7258684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:21.7259249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:21.7259711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:21.7260283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:21.7260709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:21.7261270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:21.7261723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:21.7262052Z Closing process 58671 via signal SIGTERM
2023-01-11T22:23:21.7262359Z Closing process 58672 via signal SIGTERM
2023-01-11T22:23:21.7262658Z Closing process 58673 via signal SIGTERM
2023-01-11T22:23:21.7262902Z ok (2.785s)
2023-01-11T22:23:21.7263200Z   test_function_raise (__main__.StartProcessesListTest)
2023-01-11T22:23:21.7263885Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:21.7264392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:21.7264944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:21.7265416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:21.7265990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:21.7266577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:21.7267157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:21.7267617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:21.7268025Z failed (exitcode: 1) local_rank: 0 (pid: 58803) of fn: echo2 (start_method: spawn)
2023-01-11T22:23:21.7268356Z Traceback (most recent call last):
2023-01-11T22:23:21.7268890Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2023-01-11T22:23:21.7269316Z     self._pc.join(-1)
2023-01-11T22:23:21.7269770Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 160, in join
2023-01-11T22:23:21.7270232Z     raise ProcessRaisedException(msg, error_index, failed_process.pid)
2023-01-11T22:23:21.7270685Z torch.multiprocessing.spawn.ProcessRaisedException: 
2023-01-11T22:23:21.7270960Z 
2023-01-11T22:23:21.7271192Z -- Process 0 terminated with the following error:
2023-01-11T22:23:21.7271499Z Traceback (most recent call last):
2023-01-11T22:23:21.7272510Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
2023-01-11T22:23:21.7272889Z     fn(i, *args)
2023-01-11T22:23:21.7273383Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 371, in _wrap
2023-01-11T22:23:21.7273784Z     ret = record(fn)(*args_)
2023-01-11T22:23:21.7274326Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
2023-01-11T22:23:21.7274732Z     return f(*args, **kwargs)
2023-01-11T22:23:21.7275104Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2
2023-01-11T22:23:21.7275483Z     raise RuntimeError(msg)
2023-01-11T22:23:21.7275746Z RuntimeError: hello
2023-01-11T22:23:21.7275897Z 
2023-01-11T22:23:21.7275997Z ok (2.322s)
2023-01-11T22:23:21.7276332Z   test_function_with_tensor (__main__.StartProcessesListTest) ... ok (0.002s)
2023-01-11T22:23:21.7276750Z   test_invalid_log_dir (__main__.StartProcessesListTest) ... ok (0.001s)
2023-01-11T22:23:21.7277207Z   test_multiprocess_context_close (__main__.StartProcessesListTest) ... Closing process 58869 via signal SIGTERM
2023-01-11T22:23:21.7277572Z ok (0.003s)
2023-01-11T22:23:21.7278177Z   test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesListTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn)
2023-01-11T22:23:21.7278650Z Traceback (most recent call last):
2023-01-11T22:23:21.7279166Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2023-01-11T22:23:21.7279588Z     self._pc.join(-1)
2023-01-11T22:23:21.7279913Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1104, in __call__
2023-01-11T22:23:21.7280250Z     return self._mock_call(*args, **kwargs)
2023-01-11T22:23:21.7280602Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1108, in _mock_call
2023-01-11T22:23:21.7280963Z     return self._execute_mock_call(*args, **kwargs)
2023-01-11T22:23:21.7281339Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1163, in _execute_mock_call
2023-01-11T22:23:21.7281642Z     raise effect
2023-01-11T22:23:21.7282018Z torch.multiprocessing.spawn.ProcessRaisedException: test msg
2023-01-11T22:23:21.7282379Z ok (0.004s)
2023-01-11T22:23:21.7282986Z   test_pcontext_wait (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:21.7283509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:21.7284078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:21.7284708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:21.7285026Z ok (3.287s)
2023-01-11T22:23:21.7285411Z   test_subprocess_context_close (__main__.StartProcessesListTest) ... Sending process 58903 closing signal SIGTERM
2023-01-11T22:23:21.7285779Z ok (0.003s)
2023-01-11T22:23:21.7286067Z   test_to_map (__main__.StartProcessesListTest) ... ok (0.001s)
2023-01-11T22:23:21.7286470Z   test_validate_full_rank (__main__.StartProcessesListTest) ... ok (0.001s)
2023-01-11T22:23:21.7287187Z   test_void_function (__main__.StartProcessesListTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:21.7287691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:21.7288258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:21.7288722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:21.7289301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:21.7289723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:21.7290287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:21.7290747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:21.7291029Z hello
2023-01-11T22:23:21.7291236Z world
2023-01-11T22:23:21.7291489Z Closing process 58904 via signal SIGTERM
2023-01-11T22:23:21.7291740Z ok (2.418s)
2023-01-11T22:23:21.7292063Z   test_args_env_len_mismatch (__main__.StartProcessesTest) ... ok (0.002s)
2023-01-11T22:23:21.7292468Z   test_binary_exit (__main__.StartProcessesTest) ... bar stderr from 1
2023-01-11T22:23:21.7292784Z bar stdout from 1
2023-01-11T22:23:21.7293194Z failed (exitcode: 138) local_rank: 0 (pid: 58970) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo1.py
2023-01-11T22:23:21.7293592Z ok (0.104s)
2023-01-11T22:23:21.7293924Z   test_binary_incorrect_entrypoint (__main__.StartProcessesTest) ... ok (0.002s)
2023-01-11T22:23:21.7294350Z   test_binary_raises (__main__.StartProcessesTest) ... Traceback (most recent call last):
2023-01-11T22:23:21.7294830Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py", line 22, in <module>
2023-01-11T22:23:21.7295241Z     raise RuntimeError(f"raised from {rank}")
2023-01-11T22:23:21.7295539Z RuntimeError: raised from 0
2023-01-11T22:23:21.7295776Z bar from 1
2023-01-11T22:23:21.7296189Z failed (exitcode: 1) local_rank: 0 (pid: 58973) of binary: /var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/bin/echo2.py
2023-01-11T22:23:21.7296576Z ok (0.104s)
2023-01-11T22:23:21.7297194Z   test_function_large_ret_val (__main__.StartProcessesTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:21.7297720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:21.7298287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:21.7298753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:21.7299311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:21.7299753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:21.7300316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:21.7300760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:21.7301330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:21.7301881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:21.7302454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:21.7302895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:21.7303464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:21.7303904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:21.7304452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:21.7304912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:21.7305258Z Closing process 58975 via signal SIGTERM
2023-01-11T22:23:21.7305559Z Closing process 58976 via signal SIGTERM
2023-01-11T22:23:21.7305845Z Closing process 58978 via signal SIGTERM
2023-01-11T22:23:21.7306113Z ok (2.773s)
2023-01-11T22:23:21.7306401Z   test_function_raise (__main__.StartProcessesTest)
2023-01-11T22:23:21.7307061Z run 2x copies of echo2, raise an exception on the first ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:21.7307570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:21.7308135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:21.7308597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:21.7309152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:21.7309591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:21.7310160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:21.7310605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:21.7311008Z failed (exitcode: 1) local_rank: 0 (pid: 59107) of fn: echo2 (start_method: spawn)
2023-01-11T22:23:21.7311352Z Traceback (most recent call last):
2023-01-11T22:23:21.7312447Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2023-01-11T22:23:21.7312906Z     self._pc.join(-1)
2023-01-11T22:23:21.7313378Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 160, in join
2023-01-11T22:23:21.7313836Z     raise ProcessRaisedException(msg, error_index, failed_process.pid)
2023-01-11T22:23:21.7314274Z torch.multiprocessing.spawn.ProcessRaisedException: 
2023-01-11T22:23:21.7314543Z 
2023-01-11T22:23:21.7314765Z -- Process 0 terminated with the following error:
2023-01-11T22:23:21.7315084Z Traceback (most recent call last):
2023-01-11T22:23:21.7315567Z   File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
2023-01-11T22:23:21.7315921Z     fn(i, *args)
2023-01-11T22:23:21.7316417Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 371, in _wrap
2023-01-11T22:23:21.7316802Z     ret = record(fn)(*args_)
2023-01-11T22:23:21.7317322Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
2023-01-11T22:23:21.7317726Z     return f(*args, **kwargs)
2023-01-11T22:23:21.7318117Z   File "/var/lib/jenkins/workspace/test/distributed/elastic/multiprocessing/api_test.py", line 138, in echo2
2023-01-11T22:23:21.7318474Z     raise RuntimeError(msg)
2023-01-11T22:23:21.7318737Z RuntimeError: hello
2023-01-11T22:23:21.7318899Z 
2023-01-11T22:23:21.7318997Z ok (2.321s)
2023-01-11T22:23:21.7319298Z   test_function_with_tensor (__main__.StartProcessesTest) ... ok (0.002s)
2023-01-11T22:23:21.7319887Z   test_invalid_log_dir (__main__.StartProcessesTest) ... ok (0.001s)
2023-01-11T22:23:21.7320349Z   test_multiprocess_context_close (__main__.StartProcessesTest) ... Closing process 59173 via signal SIGTERM
2023-01-11T22:23:21.7320702Z ok (0.003s)
2023-01-11T22:23:21.7321285Z   test_multiprocessing_context_poll_raises_exception (__main__.StartProcessesTest) ... failed (exitcode: -1) local_rank: 0 (pid: 123) of fn: echo0 (start_method: spawn)
2023-01-11T22:23:21.7321748Z Traceback (most recent call last):
2023-01-11T22:23:21.7322279Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 455, in _poll
2023-01-11T22:23:21.7322685Z     self._pc.join(-1)
2023-01-11T22:23:21.7323013Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1104, in __call__
2023-01-11T22:23:21.7323357Z     return self._mock_call(*args, **kwargs)
2023-01-11T22:23:21.7323709Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1108, in _mock_call
2023-01-11T22:23:21.7324058Z     return self._execute_mock_call(*args, **kwargs)
2023-01-11T22:23:21.7324431Z   File "/opt/conda/lib/python3.10/unittest/mock.py", line 1163, in _execute_mock_call
2023-01-11T22:23:21.7324752Z     raise effect
2023-01-11T22:23:21.7325107Z torch.multiprocessing.spawn.ProcessRaisedException: test msg
2023-01-11T22:23:21.7325508Z ok (0.005s)
2023-01-11T22:23:21.7326118Z   test_pcontext_wait (__main__.StartProcessesTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:21.7326631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:21.7327181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:21.7327646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:21.7327952Z ok (3.270s)
2023-01-11T22:23:21.7328313Z   test_subprocess_context_close (__main__.StartProcessesTest) ... Sending process 59207 closing signal SIGTERM
2023-01-11T22:23:21.7328676Z ok (0.003s)
2023-01-11T22:23:21.7328969Z   test_to_map (__main__.StartProcessesTest) ... ok (0.001s)
2023-01-11T22:23:21.7329351Z   test_validate_full_rank (__main__.StartProcessesTest) ... ok (0.001s)
2023-01-11T22:23:21.7330024Z   test_void_function (__main__.StartProcessesTest) ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:21.7330535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:21.7331105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:21.7331572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:21.7332128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:21.7332581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:21.7333147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:21.7333594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:21.7333896Z world
2023-01-11T22:23:21.7334103Z hello
2023-01-11T22:23:21.7334338Z Closing process 59209 via signal SIGTERM
2023-01-11T22:23:21.7334602Z ok (2.308s)
2023-01-11T22:23:21.7334890Z   test_from_str_bad_input (__main__.StdTest) ... ok (0.001s)
2023-01-11T22:23:21.7335202Z   test_from_value (__main__.StdTest) ... ok (0.001s)
2023-01-11T22:23:21.7335525Z   test_from_value_map (__main__.StdTest) ... ok (0.001s)
2023-01-11T22:23:21.7335721Z 
2023-01-11T22:23:21.7335989Z ----------------------------------------------------------------------
2023-01-11T22:23:21.7336316Z Ran 38 tests in 27.247s
2023-01-11T22:23:21.7336459Z 
2023-01-11T22:23:21.7336551Z OK
2023-01-11T22:23:21.7336750Z 
2023-01-11T22:23:21.7336874Z Generating XML reports...
2023-01-11T22:23:21.7337574Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20230111222254.xml
2023-01-11T22:23:21.7338447Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20230111222254.xml
2023-01-11T22:23:21.7339318Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20230111222254.xml
2023-01-11T22:23:21.7340129Z Generated XML report: test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20230111222254.xml
2023-01-11T22:23:21.7340487Z 
2023-01-11T22:23:21.7340924Z ##[endgroup]
2023-01-11T22:23:21.7341581Z FINISHED PRINTING LOG FILE of distributed/elastic/multiprocessing/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-multiprocessing-api_test_p63dzfto)
2023-01-11T22:23:21.7342001Z 
2023-01-11T22:23:21.7342274Z Running distributed/_tensor/test_matrix_ops ... [2023-01-11 22:23:21.722328]
2023-01-11T22:23:21.7342954Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_matrix_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:23:21.722616]
2023-01-11T22:23:53.5559588Z 
2023-01-11T22:23:53.5562532Z Expand the folded group to see the log file of distributed/_tensor/test_matrix_ops
2023-01-11T22:23:53.5564201Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_matrix_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_matrix_ops_f5_5an7s)
2023-01-11T22:23:53.5564901Z 
2023-01-11T22:23:53.5565089Z Running tests...
2023-01-11T22:23:53.5565602Z ----------------------------------------------------------------------
2023-01-11T22:23:53.5566176Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_matrix_ops
2023-01-11T22:23:53.5566661Z   test_addmm (__main__.DistMatrixOpsTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:23:53.5567143Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59309
2023-01-11T22:23:53.5567582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59310
2023-01-11T22:23:53.5570319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:53.5570791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:53.5571394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:53.5571869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:53.5572453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:53.5572882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:53.5575215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:53.5575706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:53.5576183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:53.5576677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:53.5577166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:53.5577638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:53.5578304Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:53.5578995Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:53.5579648Z ok (5.362s)
2023-01-11T22:23:53.5580148Z   test_addmm_auto_redistribute (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59390
2023-01-11T22:23:53.5580688Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59391
2023-01-11T22:23:53.5581299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:53.5581731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:53.5582303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:53.5582770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:53.5583344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:53.5583770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:53.5584344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:53.5584806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:53.5585292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:53.5585761Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:53.5586243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:53.5586720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:53.5587373Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:53.5588039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:53.5588438Z ok (3.711s)
2023-01-11T22:23:53.5588846Z   test_baddbmm (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59475
2023-01-11T22:23:53.5589352Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59476
2023-01-11T22:23:53.5589936Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:53.5590379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:53.5590946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:53.5591394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:53.5592368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:53.5592829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:53.5593413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:53.5593856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:53.5594291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:53.5594777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:53.5595242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:53.5595717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:53.5596368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:53.5597147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:53.5597587Z ok (5.717s)
2023-01-11T22:23:53.5597990Z   test_bmm (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59560
2023-01-11T22:23:53.5598489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59561
2023-01-11T22:23:53.5599090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:53.5599516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:53.5600080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:53.5600544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:53.5601099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:53.5601546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:53.5602106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:53.5602563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:53.5602976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:53.5603458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:53.5603938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:53.5604397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:53.5605042Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:53.5605732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:53.5606121Z ok (3.911s)
2023-01-11T22:23:53.5606506Z   test_mm (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59645
2023-01-11T22:23:53.5607000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59646
2023-01-11T22:23:53.5607607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:53.5608050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:53.5608603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:53.5609064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:53.5609634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:53.5610065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:53.5610629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:53.5611087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:53.5611519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:53.5611985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:53.5612462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:53.5612939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:53.5613606Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:53.5614394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:53.5614789Z ok (3.810s)
2023-01-11T22:23:53.5615187Z   test_t (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59730
2023-01-11T22:23:53.5615662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59731
2023-01-11T22:23:53.5616264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:53.5616704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:53.5617273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:53.5617721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:53.5618293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:53.5618736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:53.5619285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:53.5619750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:53.5620179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:53.5620659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:53.5621123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:53.5621599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:53.5622246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:53.5622934Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:53.5623306Z ok (3.309s)
2023-01-11T22:23:53.5623716Z   test_t_partial (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59811
2023-01-11T22:23:53.5624220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59812
2023-01-11T22:23:53.5624802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:53.5625342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:53.5625908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:53.5626368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:53.5626926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:53.5627365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:53.5627929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:53.5628369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:53.5628801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:53.5629285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:23:53.5629764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:53.5630216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:53.5630922Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:53.5631659Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:23:53.5632367Z ok (3.710s)
2023-01-11T22:23:53.5632500Z 
2023-01-11T22:23:53.5632778Z ----------------------------------------------------------------------
2023-01-11T22:23:53.5633110Z Ran 7 tests in 29.531s
2023-01-11T22:23:53.5633273Z 
2023-01-11T22:23:53.5633368Z OK
2023-01-11T22:23:53.5633501Z 
2023-01-11T22:23:53.5633605Z Generating XML reports...
2023-01-11T22:23:53.5634196Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_matrix_ops/TEST-DistMatrixOpsTest-20230111222323.xml
2023-01-11T22:23:53.5634542Z 
2023-01-11T22:23:53.5634982Z ##[endgroup]
2023-01-11T22:23:53.5635573Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_matrix_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_matrix_ops_f5_5an7s)
2023-01-11T22:23:53.5635908Z 
2023-01-11T22:23:53.5636178Z Running distributed/fsdp/test_fsdp_meta ... [2023-01-11 22:23:53.556050]
2023-01-11T22:23:53.5636847Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_meta.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:23:53.556385]
2023-01-11T22:24:27.2069982Z 
2023-01-11T22:24:27.2070540Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_meta
2023-01-11T22:24:27.2074414Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_meta (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_meta_atk4f65x)
2023-01-11T22:24:27.2075147Z 
2023-01-11T22:24:27.2075280Z Running tests...
2023-01-11T22:24:27.2075783Z ----------------------------------------------------------------------
2023-01-11T22:24:27.2076346Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_meta
2023-01-11T22:24:27.2076864Z   test_bad_arg_meta (__main__.TestFSDPWithMetaDevice) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:24:27.2078225Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59927
2023-01-11T22:24:27.2078666Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59928
2023-01-11T22:24:27.2079302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:27.2079761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:27.2080345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:27.2081539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:27.2082134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:27.2082582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:27.2083188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:27.2083641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:27.2084097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:24:27.2084599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:24:27.2085260Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:27.2086619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:27.2087777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:24:27.2088722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:24:27.2089795Z dist init r=0, world=2
2023-01-11T22:24:27.2090475Z dist init r=1, world=2
2023-01-11T22:24:27.2090805Z ok (4.948s)
2023-01-11T22:24:27.2091254Z   test_bad_arg_torchdistx (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2023-01-11T22:24:27.2091857Z   test_meta_device_with_mixed_precision (__main__.TestFSDPWithMetaDevice)
2023-01-11T22:24:27.2092988Z Tests meta device initialization with a ``param_init_fn`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60006
2023-01-11T22:24:27.2094064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60007
2023-01-11T22:24:27.2095282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:27.2095741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:27.2096307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:27.2096784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:27.2097362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:27.2097804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:27.2098356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:27.2098815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:27.2099269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:24:27.2099748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:24:27.2100402Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:27.2101103Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:27.2101632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:24:27.2102087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:24:27.2102433Z dist init r=0, world=2
2023-01-11T22:24:27.2102682Z dist init r=1, world=2
2023-01-11T22:24:27.2102899Z ok (3.310s)
2023-01-11T22:24:27.2103387Z   test_nested_model_with_meta_device_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60085
2023-01-11T22:24:27.2103967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60086
2023-01-11T22:24:27.2104575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:27.2105016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:27.2105584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:27.2106049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:27.2106604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:27.2107045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:27.2107610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:27.2108068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:27.2108500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:24:27.2109139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:24:27.2109803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:27.2110492Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:27.2110999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:24:27.2111470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:24:27.2111825Z dist init r=1, world=2
2023-01-11T22:24:27.2112591Z dist init r=0, world=2
2023-01-11T22:24:27.2112828Z ok (3.810s)
2023-01-11T22:24:27.2113324Z   test_nested_model_with_meta_device_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60168
2023-01-11T22:24:27.2113917Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60169
2023-01-11T22:24:27.2114519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:27.2114965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:27.2115535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:27.2115983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:27.2116556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:27.2117001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:27.2117568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:27.2118012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:27.2118466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:24:27.2118962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:24:27.2119603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:27.2120297Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:27.2120818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:24:27.2121288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:24:27.2121626Z dist init r=1, world=2
2023-01-11T22:24:27.2121873Z dist init r=0, world=2
2023-01-11T22:24:27.2122109Z ok (3.810s)
2023-01-11T22:24:27.2122588Z   test_nested_model_with_meta_device_reset_params_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60251
2023-01-11T22:24:27.2123175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60252
2023-01-11T22:24:27.2123785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:27.2124232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:27.2124782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:27.2125244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:27.2125817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:27.2126258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:27.2126998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:27.2127469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:27.2127922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:24:27.2128401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:24:27.2129065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:27.2129751Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:27.2130271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:24:27.2130721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:24:27.2131077Z dist init r=1, world=2
2023-01-11T22:24:27.2131324Z dist init r=0, world=2
2023-01-11T22:24:27.2131541Z ok (3.810s)
2023-01-11T22:24:27.2132030Z   test_nested_model_with_meta_device_reset_params_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60334
2023-01-11T22:24:27.2132614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60335
2023-01-11T22:24:27.2133220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:27.2133648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:27.2134216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:27.2134678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:27.2135260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:27.2135689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:27.2136254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:27.2136716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:27.2137148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:24:27.2137644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:24:27.2138297Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:27.2138988Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:27.2139501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:24:27.2139971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:24:27.2140324Z dist init r=0, world=2
2023-01-11T22:24:27.2140554Z dist init r=1, world=2
2023-01-11T22:24:27.2140789Z ok (3.810s)
2023-01-11T22:24:27.2141291Z   test_nested_model_with_torchdistX_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s)
2023-01-11T22:24:27.2142021Z   test_nested_model_with_torchdistX_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2023-01-11T22:24:27.2142718Z   test_nested_model_with_torchdistX_init_fn_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s)
2023-01-11T22:24:27.2143539Z   test_nested_model_with_torchdistX_init_fn_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s)
2023-01-11T22:24:27.2144237Z   test_simple_model_with_meta_device_default_init (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60417
2023-01-11T22:24:27.2144801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60418
2023-01-11T22:24:27.2145393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:27.2145841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:27.2146407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:27.2146870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:27.2147430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:27.2147878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:27.2148443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:27.2148890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:27.2149339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:24:27.2149836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:24:27.2150488Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:27.2151162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:27.2151687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:24:27.2152689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:24:27.2153050Z dist init r=1, world=2
2023-01-11T22:24:27.2153280Z dist init r=0, world=2
2023-01-11T22:24:27.2153516Z ok (3.910s)
2023-01-11T22:24:27.2153989Z   test_simple_model_with_meta_device_reset_params (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60500
2023-01-11T22:24:27.2154536Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60501
2023-01-11T22:24:27.2155156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:27.2155604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:27.2156173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:27.2156635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:27.2157209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:27.2157649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:27.2158199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:27.2158713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:27.2159166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:24:27.2159660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:24:27.2160306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:27.2161173Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:27.2161726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:24:27.2162202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:24:27.2162540Z dist init r=0, world=2
2023-01-11T22:24:27.2162788Z dist init r=1, world=2
2023-01-11T22:24:27.2163025Z ok (3.910s)
2023-01-11T22:24:27.2163488Z   test_simple_model_with_torchdistX_default_init (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2023-01-11T22:24:27.2164177Z   test_simple_model_with_torchdistX_init_fn (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2023-01-11T22:24:27.2164537Z 
2023-01-11T22:24:27.2164811Z ----------------------------------------------------------------------
2023-01-11T22:24:27.2165146Z Ran 15 tests in 31.324s
2023-01-11T22:24:27.2165308Z 
2023-01-11T22:24:27.2165397Z OK (skipped=7)
2023-01-11T22:24:27.2165549Z 
2023-01-11T22:24:27.2165672Z Generating XML reports...
2023-01-11T22:24:27.2166278Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20230111222355.xml
2023-01-11T22:24:27.2166645Z 
2023-01-11T22:24:27.2166969Z ##[endgroup]
2023-01-11T22:24:27.2167557Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_meta (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_meta_atk4f65x)
2023-01-11T22:24:27.2167905Z 
2023-01-11T22:24:27.2168164Z Running distributed/fsdp/test_fsdp_comm ... [2023-01-11 22:24:27.207232]
2023-01-11T22:24:27.2168833Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:24:27.207503]
2023-01-11T22:25:02.7588361Z 
2023-01-11T22:25:02.7588867Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_comm
2023-01-11T22:25:02.7591551Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_6l11ezhz)
2023-01-11T22:25:02.7592239Z 
2023-01-11T22:25:02.7592371Z Running tests...
2023-01-11T22:25:02.7592874Z ----------------------------------------------------------------------
2023-01-11T22:25:02.7594895Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm
2023-01-11T22:25:02.7595437Z   test_communication_nested_model_False_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication)
2023-01-11T22:25:02.7596061Z Tests FSDP's communication cost in terms of calls to collective ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:25:02.7596547Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60618
2023-01-11T22:25:02.7597002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60619
2023-01-11T22:25:02.7597628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:02.7598098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:02.7598677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:02.7599146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:02.7599712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:02.7600159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:02.7600733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:02.7601179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:02.7601989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:25:02.7602497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:25:02.7603163Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:02.7603835Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:02.7604351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:25:02.7604822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:25:02.7608979Z dist init r=1, world=2
2023-01-11T22:25:02.7609256Z dist init r=0, world=2
2023-01-11T22:25:02.7609500Z ok (5.718s)
2023-01-11T22:25:02.7609950Z   test_communication_nested_model_False_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2023-01-11T22:25:02.7610740Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60701
2023-01-11T22:25:02.7611573Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60702
2023-01-11T22:25:02.7612896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:02.7613472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:02.7614038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:02.7614508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:02.7615085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:02.7615599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:02.7616296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:02.7617142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:02.7618032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:25:02.7619115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:25:02.7619801Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:02.7620489Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:02.7621011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:25:02.7621474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:25:02.7621832Z dist init r=0, world=2
2023-01-11T22:25:02.7622081Z dist init r=1, world=2
2023-01-11T22:25:02.7622317Z ok (4.011s)
2023-01-11T22:25:02.7622682Z   test_communication_nested_model_False_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication)
2023-01-11T22:25:02.7623454Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60784
2023-01-11T22:25:02.7623982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60785
2023-01-11T22:25:02.7624582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:02.7625030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:02.7625582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:02.7626250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:02.7626833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:02.7627272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:02.7627821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:02.7628282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:02.7628727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:25:02.7629198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:25:02.7629852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:02.7630546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:02.7631061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:25:02.7631512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:25:02.7632401Z dist init r=1, world=2
2023-01-11T22:25:02.7632706Z dist init r=0, world=2
2023-01-11T22:25:02.7632930Z ok (4.011s)
2023-01-11T22:25:02.7633354Z   test_communication_nested_model_False_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2023-01-11T22:25:02.7634106Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60867
2023-01-11T22:25:02.7634634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60868
2023-01-11T22:25:02.7635235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:02.7635682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:02.7636250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:02.7636717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:02.7637275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:02.7637714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:02.7638280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:02.7638721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:02.7639168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:25:02.7639669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:25:02.7640319Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:02.7640984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:02.7641502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:25:02.7641969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:25:02.7642324Z dist init r=1, world=2
2023-01-11T22:25:02.7642554Z dist init r=0, world=2
2023-01-11T22:25:02.7642792Z ok (4.111s)
2023-01-11T22:25:02.7643167Z   test_communication_nested_model_True_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication)
2023-01-11T22:25:02.7644059Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60950
2023-01-11T22:25:02.7644597Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60951
2023-01-11T22:25:02.7645199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:02.7645626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:02.7646194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:02.7646663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:02.7647232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:02.7647652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:02.7648225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:02.7648683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:02.7649131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:25:02.7649603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:25:02.7650247Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:02.7650929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:02.7651433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:25:02.7651901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:25:02.7653162Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:25:02.7653998Z   warnings.warn(
2023-01-11T22:25:02.7655146Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:25:02.7655921Z   warnings.warn(
2023-01-11T22:25:02.7656152Z dist init r=0, world=2
2023-01-11T22:25:02.7656401Z dist init r=1, world=2
2023-01-11T22:25:02.7656635Z ok (3.811s)
2023-01-11T22:25:02.7657041Z   test_communication_nested_model_True_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2023-01-11T22:25:02.7657773Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61033
2023-01-11T22:25:02.7658298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61034
2023-01-11T22:25:02.7658897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:02.7659325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:02.7659896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:02.7660470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:02.7661099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:02.7661526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:02.7662095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:02.7662555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:02.7662987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:25:02.7663484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:25:02.7664140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:02.7664841Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:02.7665347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:25:02.7665815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:25:02.7667082Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:25:02.7667862Z   warnings.warn(
2023-01-11T22:25:02.7669006Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:25:02.7669761Z   warnings.warn(
2023-01-11T22:25:02.7670010Z dist init r=1, world=2
2023-01-11T22:25:02.7670257Z dist init r=0, world=2
2023-01-11T22:25:02.7670475Z ok (3.811s)
2023-01-11T22:25:02.7670851Z   test_communication_nested_model_True_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication)
2023-01-11T22:25:02.7671548Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61116
2023-01-11T22:25:02.7672659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61117
2023-01-11T22:25:02.7673272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:02.7673717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:02.7674286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:02.7674752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:02.7675311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:02.7675750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:02.7676318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:02.7676765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:02.7677324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:25:02.7677892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:25:02.7678555Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:02.7679227Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:02.7679747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:25:02.7680218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:25:02.7681482Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:25:02.7682270Z   warnings.warn(
2023-01-11T22:25:02.7683403Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:25:02.7684179Z   warnings.warn(
2023-01-11T22:25:02.7684424Z dist init r=1, world=2
2023-01-11T22:25:02.7684674Z dist init r=0, world=2
2023-01-11T22:25:02.7684892Z ok (3.911s)
2023-01-11T22:25:02.7685310Z   test_communication_nested_model_True_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2023-01-11T22:25:02.7686058Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61199
2023-01-11T22:25:02.7686573Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61200
2023-01-11T22:25:02.7687179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:02.7687624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:02.7688192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:02.7688638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:02.7689216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:02.7689668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:02.7690213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:02.7690675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:02.7691124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:25:02.7691624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:25:02.7692258Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:02.7692946Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:02.7693465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:25:02.7694053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:25:02.7695312Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:25:02.7696078Z   warnings.warn(
2023-01-11T22:25:02.7697237Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:25:02.7698017Z   warnings.warn(
2023-01-11T22:25:02.7698271Z dist init r=0, world=2
2023-01-11T22:25:02.7698500Z dist init r=1, world=2
2023-01-11T22:25:02.7698734Z ok (3.811s)
2023-01-11T22:25:02.7698882Z 
2023-01-11T22:25:02.7699156Z ----------------------------------------------------------------------
2023-01-11T22:25:02.7699468Z Ran 8 tests in 33.198s
2023-01-11T22:25:02.7699628Z 
2023-01-11T22:25:02.7699722Z OK
2023-01-11T22:25:02.7699856Z 
2023-01-11T22:25:02.7699983Z Generating XML reports...
2023-01-11T22:25:02.7700558Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm/TEST-TestCommunication-20230111222429.xml
2023-01-11T22:25:02.7700910Z 
2023-01-11T22:25:02.7701299Z ##[endgroup]
2023-01-11T22:25:02.7701887Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_6l11ezhz)
2023-01-11T22:25:02.7702240Z 
2023-01-11T22:25:02.7702507Z Running distributed/test_c10d_spawn_gloo ... [2023-01-11 22:25:02.759116]
2023-01-11T22:25:02.7703197Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:25:02.759469]
2023-01-11T22:26:16.7892899Z 
2023-01-11T22:26:16.7893668Z Expand the folded group to see the log file of distributed/test_c10d_spawn_gloo
2023-01-11T22:26:16.7894557Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_gloo (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_gloo_v40jeioz)
2023-01-11T22:26:16.7895154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfnko6o0z
2023-01-11T22:26:16.7895704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfnko6o0z/_remote_module_non_scriptable.py
2023-01-11T22:26:16.7896149Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.7896907Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelSingleProcessTest testMethod=test_cpu>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_cuda>, <__main__.DistributedDataParallelSingleProcessTest testMethod=test_rnn>]>
2023-01-11T22:26:16.7897701Z test_cpu (__main__.DistributedDataParallelSingleProcessTest)
2023-01-11T22:26:16.7898159Z test_cuda (__main__.DistributedDataParallelSingleProcessTest)
2023-01-11T22:26:16.7898609Z test_rnn (__main__.DistributedDataParallelSingleProcessTest)
2023-01-11T22:26:16.7898981Z <unittest.suite.TestSuite tests=[]>
2023-01-11T22:26:16.7899306Z <unittest.suite.TestSuite tests=[]>
2023-01-11T22:26:16.7900735Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsGloo testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_gather>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsGloo testMethod=test_scatter>]>
2023-01-11T22:26:16.7903076Z test_all_gather (__main__.TestDistributedNNFunctionsGloo)
2023-01-11T22:26:16.7903710Z test_all_to_all (__main__.TestDistributedNNFunctionsGloo)
2023-01-11T22:26:16.7907140Z test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo)
2023-01-11T22:26:16.7907786Z test_allreduce (__main__.TestDistributedNNFunctionsGloo)
2023-01-11T22:26:16.7908324Z test_broadcast (__main__.TestDistributedNNFunctionsGloo)
2023-01-11T22:26:16.7908721Z test_gather (__main__.TestDistributedNNFunctionsGloo)
2023-01-11T22:26:16.7909356Z test_reduce (__main__.TestDistributedNNFunctionsGloo)
2023-01-11T22:26:16.7909758Z test_scatter (__main__.TestDistributedNNFunctionsGloo)
2023-01-11T22:26:16.7910670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.7911211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.7912317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.7913231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.7913967Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvqy_nsg1
2023-01-11T22:26:16.7914526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvqy_nsg1/_remote_module_non_scriptable.py
2023-01-11T22:26:16.7915238Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.7915446Z 
2023-01-11T22:26:16.7915558Z Running tests...
2023-01-11T22:26:16.7915991Z ----------------------------------------------------------------------
2023-01-11T22:26:16.7916811Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T22:26:16.7917502Z   test_cpu (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:26:16.7918542Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:26:16.7919301Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:26:16.7919680Z ok (0.019s)
2023-01-11T22:26:16.7919828Z 
2023-01-11T22:26:16.7920099Z ----------------------------------------------------------------------
2023-01-11T22:26:16.7920831Z Ran 1 test in 0.019s
2023-01-11T22:26:16.7921000Z 
2023-01-11T22:26:16.7921092Z OK
2023-01-11T22:26:16.7921207Z 
2023-01-11T22:26:16.7921332Z Generating XML reports...
2023-01-11T22:26:16.7922307Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20230111222509.xml
2023-01-11T22:26:16.7923185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.7923827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.7924410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.7925159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.7925626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_kh249cz
2023-01-11T22:26:16.7926403Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_kh249cz/_remote_module_non_scriptable.py
2023-01-11T22:26:16.7926836Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.7927031Z 
2023-01-11T22:26:16.7927140Z Running tests...
2023-01-11T22:26:16.7927795Z ----------------------------------------------------------------------
2023-01-11T22:26:16.7928345Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T22:26:16.7929552Z   test_cuda (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:26:16.7930645Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:26:16.7931161Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:26:16.7931606Z ok (0.463s)
2023-01-11T22:26:16.7931881Z 
2023-01-11T22:26:16.7932197Z ----------------------------------------------------------------------
2023-01-11T22:26:16.7932505Z Ran 1 test in 0.463s
2023-01-11T22:26:16.7932666Z 
2023-01-11T22:26:16.7932759Z OK
2023-01-11T22:26:16.7932974Z 
2023-01-11T22:26:16.7933209Z Generating XML reports...
2023-01-11T22:26:16.7933984Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20230111222512.xml
2023-01-11T22:26:16.7935041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.7935498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.7936343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.7936816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.7937519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphhfzkchx
2023-01-11T22:26:16.7938076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphhfzkchx/_remote_module_non_scriptable.py
2023-01-11T22:26:16.7938645Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.7938965Z 
2023-01-11T22:26:16.7939057Z Running tests...
2023-01-11T22:26:16.7939468Z ----------------------------------------------------------------------
2023-01-11T22:26:16.7940283Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T22:26:16.7940914Z   test_rnn (__main__.DistributedDataParallelSingleProcessTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:26:16.7941957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:26:16.7942482Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:26:16.7943096Z ok (1.244s)
2023-01-11T22:26:16.7943248Z 
2023-01-11T22:26:16.7943500Z ----------------------------------------------------------------------
2023-01-11T22:26:16.7943828Z Ran 1 test in 1.244s
2023-01-11T22:26:16.7944103Z 
2023-01-11T22:26:16.7944271Z OK
2023-01-11T22:26:16.7944480Z 
2023-01-11T22:26:16.7944607Z Generating XML reports...
2023-01-11T22:26:16.7945305Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20230111222516.xml
2023-01-11T22:26:16.7946366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.7946957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.7947646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.7948190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.7948856Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwh2bbo37
2023-01-11T22:26:16.7949400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwh2bbo37/_remote_module_non_scriptable.py
2023-01-11T22:26:16.7950084Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.7950282Z 
2023-01-11T22:26:16.7950390Z Running tests...
2023-01-11T22:26:16.7951048Z ----------------------------------------------------------------------
2023-01-11T22:26:16.7951756Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T22:26:16.7953017Z   test_all_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61468
2023-01-11T22:26:16.7953565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61469
2023-01-11T22:26:16.7954468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.7954899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.7955745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.7956217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.7957081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.7957517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.7958369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.7958840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.7959406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt1cpde0i
2023-01-11T22:26:16.7960092Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt1cpde0i/_remote_module_non_scriptable.py
2023-01-11T22:26:16.7960680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxl0kp5d4
2023-01-11T22:26:16.7961429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxl0kp5d4/_remote_module_non_scriptable.py
2023-01-11T22:26:16.7961836Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.7962266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:16.7962913Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.7963297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:16.7963788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:26:16.7964277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:26:16.7965156Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:16.7965877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:16.7966267Z ok (4.025s)
2023-01-11T22:26:16.7966413Z 
2023-01-11T22:26:16.7966727Z ----------------------------------------------------------------------
2023-01-11T22:26:16.7967040Z Ran 1 test in 4.025s
2023-01-11T22:26:16.7967204Z 
2023-01-11T22:26:16.7967297Z OK
2023-01-11T22:26:16.7967431Z 
2023-01-11T22:26:16.7967554Z Generating XML reports...
2023-01-11T22:26:16.7968192Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111222520.xml
2023-01-11T22:26:16.7969024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.7969667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.7970245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.7970694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.7971157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjwo3onc5
2023-01-11T22:26:16.7971801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjwo3onc5/_remote_module_non_scriptable.py
2023-01-11T22:26:16.7972298Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.7972492Z 
2023-01-11T22:26:16.7972582Z Running tests...
2023-01-11T22:26:16.7972986Z ----------------------------------------------------------------------
2023-01-11T22:26:16.7973524Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T22:26:16.7974069Z   test_all_to_all (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61584
2023-01-11T22:26:16.7974604Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61585
2023-01-11T22:26:16.7975208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.7975655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.7976283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.7976988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.7977572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.7978010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.7978563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.7979019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.7979479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv44bj67a
2023-01-11T22:26:16.7979998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv44bj67a/_remote_module_non_scriptable.py
2023-01-11T22:26:16.7980524Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu2ffacj3
2023-01-11T22:26:16.7981066Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu2ffacj3/_remote_module_non_scriptable.py
2023-01-11T22:26:16.7981488Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.7981868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:16.7982256Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.7982651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:16.7983118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:26:16.7983605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:26:16.7984259Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:16.7984944Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:16.7985324Z ok (4.124s)
2023-01-11T22:26:16.7985470Z 
2023-01-11T22:26:16.7985736Z ----------------------------------------------------------------------
2023-01-11T22:26:16.7986055Z Ran 1 test in 4.124s
2023-01-11T22:26:16.7986213Z 
2023-01-11T22:26:16.7986287Z OK
2023-01-11T22:26:16.7986419Z 
2023-01-11T22:26:16.7986540Z Generating XML reports...
2023-01-11T22:26:16.7987174Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111222528.xml
2023-01-11T22:26:16.7987908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.7988336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.7988900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.7989451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.7990002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ftruhks
2023-01-11T22:26:16.7990797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ftruhks/_remote_module_non_scriptable.py
2023-01-11T22:26:16.7991225Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.7991419Z 
2023-01-11T22:26:16.7991526Z Running tests...
2023-01-11T22:26:16.7992215Z ----------------------------------------------------------------------
2023-01-11T22:26:16.7992772Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T22:26:16.7993350Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61700
2023-01-11T22:26:16.7993898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61701
2023-01-11T22:26:16.7994490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.7994941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.7995508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.7995950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.7996524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.7996966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.7997529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.7997970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.7998432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmyeg239m
2023-01-11T22:26:16.7998990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmyeg239m/_remote_module_non_scriptable.py
2023-01-11T22:26:16.7999510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphzf4snrc
2023-01-11T22:26:16.8000055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphzf4snrc/_remote_module_non_scriptable.py
2023-01-11T22:26:16.8000488Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.8000894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:16.8001268Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.8001672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:16.8002157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:26:16.8002637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:26:16.8003308Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:16.8004002Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:16.8004391Z ok (4.024s)
2023-01-11T22:26:16.8004536Z 
2023-01-11T22:26:16.8004783Z ----------------------------------------------------------------------
2023-01-11T22:26:16.8005106Z Ran 1 test in 4.025s
2023-01-11T22:26:16.8005265Z 
2023-01-11T22:26:16.8005356Z OK
2023-01-11T22:26:16.8005485Z 
2023-01-11T22:26:16.8005592Z Generating XML reports...
2023-01-11T22:26:16.8006233Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111222535.xml
2023-01-11T22:26:16.8006970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.8007534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.8008151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.8008627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.8009093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnu91v5n_
2023-01-11T22:26:16.8009643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnu91v5n_/_remote_module_non_scriptable.py
2023-01-11T22:26:16.8010056Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.8010253Z 
2023-01-11T22:26:16.8010362Z Running tests...
2023-01-11T22:26:16.8010768Z ----------------------------------------------------------------------
2023-01-11T22:26:16.8011290Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T22:26:16.8011876Z   test_allreduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61816
2023-01-11T22:26:16.8012426Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61817
2023-01-11T22:26:16.8013032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.8013462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.8030287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.8030764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.8031351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.8031806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.8033322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.8033834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.8034313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl5ayzgir
2023-01-11T22:26:16.8034859Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl5ayzgir/_remote_module_non_scriptable.py
2023-01-11T22:26:16.8035381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa9_b9h8_
2023-01-11T22:26:16.8035918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa9_b9h8_/_remote_module_non_scriptable.py
2023-01-11T22:26:16.8036342Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.8036747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:16.8037120Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.8037514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:16.8038007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:26:16.8038488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:26:16.8039161Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:16.8039853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:16.8040246Z ok (4.024s)
2023-01-11T22:26:16.8040376Z 
2023-01-11T22:26:16.8040642Z ----------------------------------------------------------------------
2023-01-11T22:26:16.8040966Z Ran 1 test in 4.024s
2023-01-11T22:26:16.8041127Z 
2023-01-11T22:26:16.8041217Z OK
2023-01-11T22:26:16.8041347Z 
2023-01-11T22:26:16.8041453Z Generating XML reports...
2023-01-11T22:26:16.8042091Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111222542.xml
2023-01-11T22:26:16.8043084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.8043545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.8044102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.8044571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.8045038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_uzetyg
2023-01-11T22:26:16.8045567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_uzetyg/_remote_module_non_scriptable.py
2023-01-11T22:26:16.8045998Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.8046195Z 
2023-01-11T22:26:16.8046301Z Running tests...
2023-01-11T22:26:16.8046699Z ----------------------------------------------------------------------
2023-01-11T22:26:16.8047228Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T22:26:16.8047802Z   test_broadcast (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61932
2023-01-11T22:26:16.8048344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61933
2023-01-11T22:26:16.8048930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.8049374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.8049939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.8050401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.8050954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.8051403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.8051959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.8052418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.8052867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcfus3wp2
2023-01-11T22:26:16.8053411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcfus3wp2/_remote_module_non_scriptable.py
2023-01-11T22:26:16.8053948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc8vxult8
2023-01-11T22:26:16.8054469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc8vxult8/_remote_module_non_scriptable.py
2023-01-11T22:26:16.8054894Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.8055298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:16.8055694Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.8056073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:16.8056557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:26:16.8057048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:26:16.8057688Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:16.8058376Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:16.8058770Z ok (4.024s)
2023-01-11T22:26:16.8058914Z 
2023-01-11T22:26:16.8059178Z ----------------------------------------------------------------------
2023-01-11T22:26:16.8059482Z Ran 1 test in 4.024s
2023-01-11T22:26:16.8059709Z 
2023-01-11T22:26:16.8059805Z OK
2023-01-11T22:26:16.8059979Z 
2023-01-11T22:26:16.8060106Z Generating XML reports...
2023-01-11T22:26:16.8060732Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111222550.xml
2023-01-11T22:26:16.8061469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.8061919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.8062485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.8062935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.8063399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv6igf5xt
2023-01-11T22:26:16.8063948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv6igf5xt/_remote_module_non_scriptable.py
2023-01-11T22:26:16.8064369Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.8064567Z 
2023-01-11T22:26:16.8064672Z Running tests...
2023-01-11T22:26:16.8065071Z ----------------------------------------------------------------------
2023-01-11T22:26:16.8065608Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T22:26:16.8066162Z   test_gather (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62048
2023-01-11T22:26:16.8066754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62049
2023-01-11T22:26:16.8067364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.8067812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.8068365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.8068835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.8069409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.8069827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.8070394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.8070851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.8071313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgwynph0c
2023-01-11T22:26:16.8072056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgwynph0c/_remote_module_non_scriptable.py
2023-01-11T22:26:16.8072625Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsgpbefk7
2023-01-11T22:26:16.8073174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsgpbefk7/_remote_module_non_scriptable.py
2023-01-11T22:26:16.8073591Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.8074003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:16.8074391Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.8074794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:16.8075263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:26:16.8075759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:26:16.8076430Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:16.8077106Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:16.8077606Z ok (4.025s)
2023-01-11T22:26:16.8077810Z 
2023-01-11T22:26:16.8078085Z ----------------------------------------------------------------------
2023-01-11T22:26:16.8078409Z Ran 1 test in 4.025s
2023-01-11T22:26:16.8078550Z 
2023-01-11T22:26:16.8078642Z OK
2023-01-11T22:26:16.8078774Z 
2023-01-11T22:26:16.8078896Z Generating XML reports...
2023-01-11T22:26:16.8079536Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111222557.xml
2023-01-11T22:26:16.8080258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.8080706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.8081273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.8081742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.8082197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7obnoaw9
2023-01-11T22:26:16.8082741Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7obnoaw9/_remote_module_non_scriptable.py
2023-01-11T22:26:16.8083167Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.8083362Z 
2023-01-11T22:26:16.8083467Z Running tests...
2023-01-11T22:26:16.8083848Z ----------------------------------------------------------------------
2023-01-11T22:26:16.8084383Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T22:26:16.8084957Z   test_reduce (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62164
2023-01-11T22:26:16.8085482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62165
2023-01-11T22:26:16.8086083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.8086539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.8087111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.8087566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.8088138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.8088580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.8089125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.8089587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.8090047Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu09mxxly
2023-01-11T22:26:16.8090591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu09mxxly/_remote_module_non_scriptable.py
2023-01-11T22:26:16.8091109Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ul2w_e9
2023-01-11T22:26:16.8091646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ul2w_e9/_remote_module_non_scriptable.py
2023-01-11T22:26:16.8092073Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.8092460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:16.8092850Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.8093254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:16.8093743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:26:16.8094223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:26:16.8094937Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:16.8095687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:16.8096084Z ok (4.024s)
2023-01-11T22:26:16.8096213Z 
2023-01-11T22:26:16.8096481Z ----------------------------------------------------------------------
2023-01-11T22:26:16.8096810Z Ran 1 test in 4.024s
2023-01-11T22:26:16.8096971Z 
2023-01-11T22:26:16.8097066Z OK
2023-01-11T22:26:16.8097181Z 
2023-01-11T22:26:16.8097304Z Generating XML reports...
2023-01-11T22:26:16.8097936Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111222604.xml
2023-01-11T22:26:16.8098669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.8099118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.8099679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.8100141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.8100605Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi2b67q53
2023-01-11T22:26:16.8101128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi2b67q53/_remote_module_non_scriptable.py
2023-01-11T22:26:16.8101556Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.8101752Z 
2023-01-11T22:26:16.8101857Z Running tests...
2023-01-11T22:26:16.8102255Z ----------------------------------------------------------------------
2023-01-11T22:26:16.8102776Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_gloo
2023-01-11T22:26:16.8103347Z   test_scatter (__main__.TestDistributedNNFunctionsGloo) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62280
2023-01-11T22:26:16.8103893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62281
2023-01-11T22:26:16.8104478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.8104922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.8105482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.8105943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.8106494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:16.8106936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:16.8107495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:16.8107946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:16.8108407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp37pt_61h
2023-01-11T22:26:16.8108947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp37pt_61h/_remote_module_non_scriptable.py
2023-01-11T22:26:16.8109479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2vsrpxeo
2023-01-11T22:26:16.8110001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2vsrpxeo/_remote_module_non_scriptable.py
2023-01-11T22:26:16.8110429Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.8110833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:16.8111204Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:16.8111601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:16.8112362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:26:16.8113018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:26:16.8113679Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:16.8114366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:16.8114757Z ok (4.025s)
2023-01-11T22:26:16.8114905Z 
2023-01-11T22:26:16.8115170Z ----------------------------------------------------------------------
2023-01-11T22:26:16.8115474Z Ran 1 test in 4.025s
2023-01-11T22:26:16.8115631Z 
2023-01-11T22:26:16.8115723Z OK
2023-01-11T22:26:16.8115854Z 
2023-01-11T22:26:16.8115975Z Generating XML reports...
2023-01-11T22:26:16.8116599Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111222612.xml
2023-01-11T22:26:16.8116994Z 
2023-01-11T22:26:16.8117459Z ##[endgroup]
2023-01-11T22:26:16.8118050Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_gloo (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_gloo_v40jeioz)
2023-01-11T22:26:16.8118394Z 
2023-01-11T22:26:16.8118649Z Running distributed/test_c10d_common ... [2023-01-11 22:26:16.789562]
2023-01-11T22:26:16.8119319Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_common.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:26:16.789836]
2023-01-11T22:27:25.8080104Z 
2023-01-11T22:27:25.8080887Z Expand the folded group to see the log file of distributed/test_c10d_common
2023-01-11T22:27:25.8081897Z ##[group]PRINTING LOG FILE of distributed/test_c10d_common (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_common_wm207ye2)
2023-01-11T22:27:25.8082449Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_debug_level>]>
2023-01-11T22:27:25.8082852Z test_debug_level (__main__.CommTest)
2023-01-11T22:27:25.8083664Z <unittest.suite.TestSuite tests=[<__main__.ComputeBucketAssignmentTest testMethod=test_multi_limit_multi_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_multi_limit_single_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_single_limit_multi_dtype>, <__main__.ComputeBucketAssignmentTest testMethod=test_single_limit_single_dtype>]>
2023-01-11T22:27:25.8084504Z test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest)
2023-01-11T22:27:25.8084927Z test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest)
2023-01-11T22:27:25.8085360Z test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest)
2023-01-11T22:27:25.8085792Z test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest)
2023-01-11T22:27:25.8086677Z <unittest.suite.TestSuite tests=[<__main__.PythonProcessGroupExtensionTest testMethod=test_backend_class_attr>, <__main__.PythonProcessGroupExtensionTest testMethod=test_collectives>, <__main__.PythonProcessGroupExtensionTest testMethod=test_get_backend_name>, <__main__.PythonProcessGroupExtensionTest testMethod=test_send_recv>]>
2023-01-11T22:27:25.8087509Z test_backend_class_attr (__main__.PythonProcessGroupExtensionTest)
2023-01-11T22:27:25.8087945Z test_collectives (__main__.PythonProcessGroupExtensionTest)
2023-01-11T22:27:25.8088386Z test_get_backend_name (__main__.PythonProcessGroupExtensionTest)
2023-01-11T22:27:25.8088812Z test_send_recv (__main__.PythonProcessGroupExtensionTest)
2023-01-11T22:27:25.8089486Z <unittest.suite.TestSuite tests=[<__main__.ReduceOpTest testMethod=test_op_isinstance_of_reduceop>, <__main__.ReduceOpTest testMethod=test_reduceop_copyable>, <__main__.ReduceOpTest testMethod=test_reduceop_equal>, <__main__.ReduceOpTest testMethod=test_reduceop_pickle>]>
2023-01-11T22:27:25.8090135Z test_op_isinstance_of_reduceop (__main__.ReduceOpTest)
2023-01-11T22:27:25.8090484Z test_reduceop_copyable (__main__.ReduceOpTest)
2023-01-11T22:27:25.8091097Z test_reduceop_equal (__main__.ReduceOpTest)
2023-01-11T22:27:25.8091496Z test_reduceop_pickle (__main__.ReduceOpTest)
2023-01-11T22:27:25.8092182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8092642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8094976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8095949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8096464Z 
2023-01-11T22:27:25.8096664Z Running tests...
2023-01-11T22:27:25.8097555Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8098310Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T22:27:25.8098760Z   test_debug_level (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:27:25.8099225Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62429
2023-01-11T22:27:25.8099674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62430
2023-01-11T22:27:25.8100293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8100924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8101546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8102018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8102603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8103032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8103609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8104086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8104504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:27:25.8104983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:27:25.8105327Z ok (3.946s)
2023-01-11T22:27:25.8105480Z 
2023-01-11T22:27:25.8105752Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8106062Z Ran 1 test in 3.946s
2023-01-11T22:27:25.8106221Z 
2023-01-11T22:27:25.8106314Z OK
2023-01-11T22:27:25.8106448Z 
2023-01-11T22:27:25.8106574Z Generating XML reports...
2023-01-11T22:27:25.8107102Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20230111222620.xml
2023-01-11T22:27:25.8107763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8108217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8108786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8109235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8109463Z 
2023-01-11T22:27:25.8109575Z Running tests...
2023-01-11T22:27:25.8109982Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8110490Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T22:27:25.8111007Z   test_multi_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:27:25.8111387Z ok (1.686s)
2023-01-11T22:27:25.8111534Z 
2023-01-11T22:27:25.8111801Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8112856Z Ran 1 test in 1.687s
2023-01-11T22:27:25.8113197Z 
2023-01-11T22:27:25.8113292Z OK
2023-01-11T22:27:25.8113485Z 
2023-01-11T22:27:25.8113617Z Generating XML reports...
2023-01-11T22:27:25.8114238Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20230111222626.xml
2023-01-11T22:27:25.8114966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8115414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8115985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8116435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8116663Z 
2023-01-11T22:27:25.8116773Z Running tests...
2023-01-11T22:27:25.8117171Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8117686Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T22:27:25.8118214Z   test_multi_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:27:25.8118597Z ok (1.673s)
2023-01-11T22:27:25.8118746Z 
2023-01-11T22:27:25.8119010Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8119315Z Ran 1 test in 1.673s
2023-01-11T22:27:25.8119474Z 
2023-01-11T22:27:25.8119567Z OK
2023-01-11T22:27:25.8119699Z 
2023-01-11T22:27:25.8119823Z Generating XML reports...
2023-01-11T22:27:25.8120420Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20230111222630.xml
2023-01-11T22:27:25.8121142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8121585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8122164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8122616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8122975Z 
2023-01-11T22:27:25.8123087Z Running tests...
2023-01-11T22:27:25.8123495Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8124001Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T22:27:25.8124522Z   test_single_limit_multi_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:27:25.8124905Z ok (1.685s)
2023-01-11T22:27:25.8125052Z 
2023-01-11T22:27:25.8125313Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8125621Z Ran 1 test in 1.685s
2023-01-11T22:27:25.8125782Z 
2023-01-11T22:27:25.8125877Z OK
2023-01-11T22:27:25.8126010Z 
2023-01-11T22:27:25.8126135Z Generating XML reports...
2023-01-11T22:27:25.8126742Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20230111222634.xml
2023-01-11T22:27:25.8127460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8127908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8128476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8128926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8129155Z 
2023-01-11T22:27:25.8129264Z Running tests...
2023-01-11T22:27:25.8129661Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8130173Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T22:27:25.8130692Z   test_single_limit_single_dtype (__main__.ComputeBucketAssignmentTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:27:25.8131192Z ok (1.636s)
2023-01-11T22:27:25.8131345Z 
2023-01-11T22:27:25.8131616Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8131919Z Ran 1 test in 1.637s
2023-01-11T22:27:25.8132079Z 
2023-01-11T22:27:25.8132172Z OK
2023-01-11T22:27:25.8132305Z 
2023-01-11T22:27:25.8132433Z Generating XML reports...
2023-01-11T22:27:25.8133029Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20230111222638.xml
2023-01-11T22:27:25.8133749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8134198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8134767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8135213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8135449Z 
2023-01-11T22:27:25.8135561Z Running tests...
2023-01-11T22:27:25.8135958Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8136487Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T22:27:25.8136999Z   test_backend_class_attr (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:27:25.8137506Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62668
2023-01-11T22:27:25.8137956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62669
2023-01-11T22:27:25.8138379Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62670
2023-01-11T22:27:25.8138812Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62671
2023-01-11T22:27:25.8139411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8139862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8140411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8140876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8141446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8141866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8142432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8142890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8143462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8143893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8144456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8144914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8145461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8145898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8146461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8146915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8147329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:27:25.8147798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:27:25.8148367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:27:25.8148835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:27:25.8149159Z ok (4.098s)
2023-01-11T22:27:25.8149305Z 
2023-01-11T22:27:25.8149576Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8149903Z Ran 1 test in 4.098s
2023-01-11T22:27:25.8150063Z 
2023-01-11T22:27:25.8150137Z OK
2023-01-11T22:27:25.8150269Z 
2023-01-11T22:27:25.8150394Z Generating XML reports...
2023-01-11T22:27:25.8151029Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20230111222642.xml
2023-01-11T22:27:25.8151763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8152667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8153261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8153724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8153952Z 
2023-01-11T22:27:25.8154042Z Running tests...
2023-01-11T22:27:25.8154447Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8154972Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T22:27:25.8155494Z   test_collectives (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:27:25.8155978Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62839
2023-01-11T22:27:25.8156430Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62840
2023-01-11T22:27:25.8156872Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62841
2023-01-11T22:27:25.8157301Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62842
2023-01-11T22:27:25.8157900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8158344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8158913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8159361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8159928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8160367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8160914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8161385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8161954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8162398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8162945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8163406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8163976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8164414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8164956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8165411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8166000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:27:25.8166627Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2023-01-11T22:27:25.8167127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:27:25.8167749Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2023-01-11T22:27:25.8168248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:27:25.8168694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:27:25.8169174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:27:25.8169663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:27:25.8170164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:27:25.8170634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:27:25.8171280Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:27:25.8172018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:27:25.8172684Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:27:25.8173358Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:27:25.8173750Z ok (5.982s)
2023-01-11T22:27:25.8173901Z 
2023-01-11T22:27:25.8174179Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8174488Z Ran 1 test in 5.982s
2023-01-11T22:27:25.8174647Z 
2023-01-11T22:27:25.8174740Z OK
2023-01-11T22:27:25.8174873Z 
2023-01-11T22:27:25.8174997Z Generating XML reports...
2023-01-11T22:27:25.8175615Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20230111222648.xml
2023-01-11T22:27:25.8176350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8176796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8177367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8177817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8178048Z 
2023-01-11T22:27:25.8178156Z Running tests...
2023-01-11T22:27:25.8178563Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8179089Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T22:27:25.8179593Z   test_get_backend_name (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:27:25.8180101Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63019
2023-01-11T22:27:25.8180548Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63020
2023-01-11T22:27:25.8180971Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63021
2023-01-11T22:27:25.8181404Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63022
2023-01-11T22:27:25.8182002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8182449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8183103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8183554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8184124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8184573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8185154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8185615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8186186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8186603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8187170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8187631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8188183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8188623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8189187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8189646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8190060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:27:25.8190530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:27:25.8190990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:27:25.8191467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:27:25.8191790Z ok (4.083s)
2023-01-11T22:27:25.8192317Z 
2023-01-11T22:27:25.8192595Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8192926Z Ran 1 test in 4.083s
2023-01-11T22:27:25.8193087Z 
2023-01-11T22:27:25.8193159Z OK
2023-01-11T22:27:25.8193295Z 
2023-01-11T22:27:25.8193421Z Generating XML reports...
2023-01-11T22:27:25.8194059Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20230111222657.xml
2023-01-11T22:27:25.8194800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8195229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8195802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8196275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8196504Z 
2023-01-11T22:27:25.8196593Z Running tests...
2023-01-11T22:27:25.8196994Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8197523Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T22:27:25.8198043Z   test_send_recv (__main__.PythonProcessGroupExtensionTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:27:25.8198522Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63190
2023-01-11T22:27:25.8198972Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63191
2023-01-11T22:27:25.8199417Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63192
2023-01-11T22:27:25.8199836Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63193
2023-01-11T22:27:25.8200586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8201042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8201614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8202063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8202638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8203079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8203625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8204086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8204660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8205113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8205657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8206115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8206682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8207119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8207665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8208122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8208552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:27:25.8209175Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2023-01-11T22:27:25.8209678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:27:25.8210300Z [W socket.cpp:601] [c10d] The client socket has failed to connect to [localhost]:6789 (errno: 99 - Cannot assign requested address).
2023-01-11T22:27:25.8210797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:27:25.8211244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:27:25.8211726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:27:25.8212218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:27:25.8212705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:27:25.8213179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:27:25.8213825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:27:25.8214511Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:27:25.8215180Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:27:25.8215860Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:27:25.8216251Z ok (6.093s)
2023-01-11T22:27:25.8216398Z 
2023-01-11T22:27:25.8216667Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8216976Z Ran 1 test in 6.093s
2023-01-11T22:27:25.8217220Z 
2023-01-11T22:27:25.8217315Z OK
2023-01-11T22:27:25.8217448Z 
2023-01-11T22:27:25.8217613Z Generating XML reports...
2023-01-11T22:27:25.8218240Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20230111222703.xml
2023-01-11T22:27:25.8218979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8219429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8220001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8220448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8220676Z 
2023-01-11T22:27:25.8220784Z Running tests...
2023-01-11T22:27:25.8221186Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8221718Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T22:27:25.8222194Z   test_op_isinstance_of_reduceop (__main__.ReduceOpTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:27:25.8222549Z ok (1.691s)
2023-01-11T22:27:25.8222696Z 
2023-01-11T22:27:25.8222959Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8223263Z Ran 1 test in 1.691s
2023-01-11T22:27:25.8223424Z 
2023-01-11T22:27:25.8223520Z OK
2023-01-11T22:27:25.8223652Z 
2023-01-11T22:27:25.8223775Z Generating XML reports...
2023-01-11T22:27:25.8224312Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20230111222711.xml
2023-01-11T22:27:25.8224988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8225438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8226006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8226460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8226694Z 
2023-01-11T22:27:25.8226804Z Running tests...
2023-01-11T22:27:25.8227204Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8227714Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T22:27:25.8228192Z   test_reduceop_copyable (__main__.ReduceOpTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:27:25.8228541Z ok (1.649s)
2023-01-11T22:27:25.8228688Z 
2023-01-11T22:27:25.8228951Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8229259Z Ran 1 test in 1.650s
2023-01-11T22:27:25.8229422Z 
2023-01-11T22:27:25.8229516Z OK
2023-01-11T22:27:25.8229647Z 
2023-01-11T22:27:25.8229771Z Generating XML reports...
2023-01-11T22:27:25.8230305Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20230111222715.xml
2023-01-11T22:27:25.8230984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8231432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8232395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8232858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8233091Z 
2023-01-11T22:27:25.8233201Z Running tests...
2023-01-11T22:27:25.8233612Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8234120Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T22:27:25.8234594Z   test_reduceop_equal (__main__.ReduceOpTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:27:25.8234936Z ok (1.666s)
2023-01-11T22:27:25.8235177Z 
2023-01-11T22:27:25.8235498Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8235810Z Ran 1 test in 1.666s
2023-01-11T22:27:25.8235970Z 
2023-01-11T22:27:25.8236064Z OK
2023-01-11T22:27:25.8236197Z 
2023-01-11T22:27:25.8236323Z Generating XML reports...
2023-01-11T22:27:25.8236865Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20230111222719.xml
2023-01-11T22:27:25.8237538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:25.8237986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:25.8238560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:25.8239009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:25.8239237Z 
2023-01-11T22:27:25.8239347Z Running tests...
2023-01-11T22:27:25.8239758Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8240267Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_common
2023-01-11T22:27:25.8240741Z   test_reduceop_pickle (__main__.ReduceOpTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:27:25.8241079Z ok (1.639s)
2023-01-11T22:27:25.8241226Z 
2023-01-11T22:27:25.8241489Z ----------------------------------------------------------------------
2023-01-11T22:27:25.8241795Z Ran 1 test in 1.639s
2023-01-11T22:27:25.8241955Z 
2023-01-11T22:27:25.8242048Z OK
2023-01-11T22:27:25.8242179Z 
2023-01-11T22:27:25.8242302Z Generating XML reports...
2023-01-11T22:27:25.8242838Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20230111222723.xml
2023-01-11T22:27:25.8243161Z 
2023-01-11T22:27:25.8243601Z ##[endgroup]
2023-01-11T22:27:25.8244173Z FINISHED PRINTING LOG FILE of distributed/test_c10d_common (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_common_wm207ye2)
2023-01-11T22:27:25.8244513Z 
2023-01-11T22:27:25.8244775Z Running distributed/_tensor/test_dtensor ... [2023-01-11 22:27:25.808480]
2023-01-11T22:27:25.8245427Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_dtensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:27:25.808732]
2023-01-11T22:28:05.2178833Z 
2023-01-11T22:28:05.2179443Z Expand the folded group to see the log file of distributed/_tensor/test_dtensor
2023-01-11T22:28:05.2180381Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_dtensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_dtensor_0_zsgp8r)
2023-01-11T22:28:05.2181305Z 
2023-01-11T22:28:05.2183179Z Running tests...
2023-01-11T22:28:05.2183732Z ----------------------------------------------------------------------
2023-01-11T22:28:05.2184295Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_dtensor
2023-01-11T22:28:05.2185338Z   test_device_mesh_nd (__main__.DTensorMeshTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:28:05.2185871Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63506
2023-01-11T22:28:05.2186328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63507
2023-01-11T22:28:05.2186978Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63508
2023-01-11T22:28:05.2187417Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63509
2023-01-11T22:28:05.2187861Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 63510
2023-01-11T22:28:05.2188302Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 63511
2023-01-11T22:28:05.2190651Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 63512
2023-01-11T22:28:05.2191720Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 63513
2023-01-11T22:28:05.2193606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2194244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2194845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2195328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2196038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2196920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2197795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2198725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2199812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2200645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2201721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2202525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2203501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2204224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2205189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2205932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2206796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2207254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2207999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2208447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2209021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2209456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2209999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2210460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2211029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2211463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2212017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2212476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2213045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2213459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2214016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2214471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2214905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:28:05.2215359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:28:05.2215995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T22:28:05.2216471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T22:28:05.2216931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:28:05.2217374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T22:28:05.2217826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T22:28:05.2218286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:28:05.2218656Z skip: Need at least 8 CUDA devices (4.279s)
2023-01-11T22:28:05.2219113Z   test_dtensor_2d_mesh (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63778
2023-01-11T22:28:05.2219616Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63779
2023-01-11T22:28:05.2220066Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63780
2023-01-11T22:28:05.2220490Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63781
2023-01-11T22:28:05.2220916Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 63782
2023-01-11T22:28:05.2221352Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 63783
2023-01-11T22:28:05.2221768Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 63784
2023-01-11T22:28:05.2222196Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 63785
2023-01-11T22:28:05.2222804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2223250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2223804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2224275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2224846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2225268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2225835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2226295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2226866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2227282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2227849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2228315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2228886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2229303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2229866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2230324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2230875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2231314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2232182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2232770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2233427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2233906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2234493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2234930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2235499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2235941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2236510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2236950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2237531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2237969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2238531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2238974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2239410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T22:28:05.2239880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:28:05.2240325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:28:05.2240777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:28:05.2241242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T22:28:05.2241703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:28:05.2242147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T22:28:05.2242600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T22:28:05.2242987Z skip: Need at least 8 CUDA devices (2.514s)
2023-01-11T22:28:05.2243460Z   test_dtensor_api_device_mesh_context_manager (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64050
2023-01-11T22:28:05.2244000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64051
2023-01-11T22:28:05.2244438Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64052
2023-01-11T22:28:05.2244874Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64053
2023-01-11T22:28:05.2245302Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 64054
2023-01-11T22:28:05.2245729Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 64055
2023-01-11T22:28:05.2246158Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 64056
2023-01-11T22:28:05.2246572Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 64057
2023-01-11T22:28:05.2247175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2247621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2248188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2248636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2249207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2249750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2250329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2250770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2251341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2251779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2252323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2252779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2253348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2253790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2254338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2254792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2255358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2255773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2256335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2256789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2257353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2257771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2258334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2258789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2259354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2259768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2260328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2260779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2261324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2261765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2262338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2262796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2263205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:28:05.2263671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:28:05.2264132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T22:28:05.2264575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T22:28:05.2265031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:28:05.2265487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:28:05.2265993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T22:28:05.2266484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T22:28:05.2266868Z skip: Need at least 8 CUDA devices (2.515s)
2023-01-11T22:28:05.2267358Z   test_dtensor_device_mesh_device_conversion (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64322
2023-01-11T22:28:05.2267896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64323
2023-01-11T22:28:05.2268318Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64324
2023-01-11T22:28:05.2268754Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64325
2023-01-11T22:28:05.2269195Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 64326
2023-01-11T22:28:05.2269613Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 64327
2023-01-11T22:28:05.2270048Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 64328
2023-01-11T22:28:05.2270480Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 64329
2023-01-11T22:28:05.2271083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2271512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2272348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2272817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2273380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2273821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2274391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2274904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2275463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2275899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2276466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2276906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2277474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2277914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2278476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2278918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2279490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2279926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2280487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2280928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2281499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2281935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2282477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2283038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2283671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2284122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2284669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2285128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2285698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2286113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2286677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2287130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2287568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T22:28:05.2288018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T22:28:05.2288471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:28:05.2288932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:28:05.2289391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:28:05.2289836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T22:28:05.2290288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:28:05.2290742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T22:28:05.2291111Z skip: Need at least 8 CUDA devices (2.515s)
2023-01-11T22:28:05.2291596Z   test_dtensor_spec_local_shard_offset (__main__.DTensorMeshTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64594
2023-01-11T22:28:05.2292120Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64595
2023-01-11T22:28:05.2292561Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 64596
2023-01-11T22:28:05.2292982Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 64597
2023-01-11T22:28:05.2293409Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 64598
2023-01-11T22:28:05.2293844Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 64599
2023-01-11T22:28:05.2294258Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 64600
2023-01-11T22:28:05.2294683Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 64601
2023-01-11T22:28:05.2295293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2295743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2296293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2296759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2297336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2297759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2298325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2298786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2299354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2299901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2300480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2300939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2301506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2301927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2302488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2302943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2303492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2303941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2304507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2304962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2305512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2305949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2306512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2306952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2307524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2307960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2308533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2308969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2309539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2309974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2310532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2310969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2311402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T22:28:05.2312222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T22:28:05.2312698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:28:05.2313156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T22:28:05.2313620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T22:28:05.2314079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:28:05.2314518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:28:05.2314966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:28:05.2315349Z skip: Need at least 8 CUDA devices (2.615s)
2023-01-11T22:28:05.2315788Z   test_dtensor_constructor (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64866
2023-01-11T22:28:05.2316299Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64867
2023-01-11T22:28:05.2317088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2317541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2318095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2318559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2319131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2319551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2320120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2320573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2321018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:28:05.2321482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:28:05.2321959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:28:05.2322439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:28:05.2323090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:05.2323763Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:05.2324154Z ok (3.209s)
2023-01-11T22:28:05.2324568Z   test_dtensor_properties (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64945
2023-01-11T22:28:05.2325068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64946
2023-01-11T22:28:05.2325670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2326114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2326683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2327134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2327706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2328149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2328716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2329162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2329603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:28:05.2330072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:28:05.2330542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:28:05.2331033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:28:05.2331686Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:05.2332368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:05.2332740Z ok (3.209s)
2023-01-11T22:28:05.2333167Z   test_dtensor_spec_read_only_after_set (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65024
2023-01-11T22:28:05.2333805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65025
2023-01-11T22:28:05.2334396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2334843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2335411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2335871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2336425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2336866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2337430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2337897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2338314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:28:05.2338797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:28:05.2339279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:28:05.2339738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:28:05.2340390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:05.2341081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:05.2341470Z ok (3.309s)
2023-01-11T22:28:05.2341858Z   test_dtensor_stride (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65103
2023-01-11T22:28:05.2342359Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65104
2023-01-11T22:28:05.2342957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2343406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2343959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2344428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2344999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2345422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2345986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2346451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2346886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:28:05.2347350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:28:05.2347827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:28:05.2348305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:28:05.2348932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:05.2349619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:05.2350010Z ok (3.210s)
2023-01-11T22:28:05.2350467Z   test_from_local (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65182
2023-01-11T22:28:05.2350996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65183
2023-01-11T22:28:05.2351601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2352245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2352824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2353272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2353843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2354282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2354836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2355298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2355731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:28:05.2356212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:28:05.2356671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:28:05.2357146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:28:05.2357796Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:05.2358480Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:05.2358850Z ok (3.210s)
2023-01-11T22:28:05.2359276Z   test_from_local_then_to_local (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65265
2023-01-11T22:28:05.2359783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65266
2023-01-11T22:28:05.2360367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2360809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2361378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2361844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2362401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2362837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2363412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2363857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2364292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:28:05.2364774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:28:05.2365253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:28:05.2365712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:28:05.2366361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:05.2367045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:05.2367539Z ok (3.209s)
2023-01-11T22:28:05.2367983Z   test_to_local (__main__.DTensorTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65348
2023-01-11T22:28:05.2368484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65349
2023-01-11T22:28:05.2369092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2369522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2370091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2370555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2371126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:05.2371547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:05.2372116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:05.2372576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:05.2372990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:28:05.2373474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:28:05.2373952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:28:05.2374428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:28:05.2375109Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:05.2375794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:05.2376190Z ok (3.310s)
2023-01-11T22:28:05.2376339Z 
2023-01-11T22:28:05.2376610Z ----------------------------------------------------------------------
2023-01-11T22:28:05.2376925Z Ran 12 tests in 37.105s
2023-01-11T22:28:05.2377089Z 
2023-01-11T22:28:05.2377196Z OK (skipped=5)
2023-01-11T22:28:05.2377347Z 
2023-01-11T22:28:05.2377469Z Generating XML reports...
2023-01-11T22:28:05.2378009Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorTest-20230111222727.xml
2023-01-11T22:28:05.2378725Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorMeshTest-20230111222727.xml
2023-01-11T22:28:05.2379060Z 
2023-01-11T22:28:05.2379519Z ##[endgroup]
2023-01-11T22:28:05.2380112Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_dtensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_dtensor_0_zsgp8r)
2023-01-11T22:28:05.2380437Z 
2023-01-11T22:28:05.2380705Z Running distributed/test_pg_wrapper ... [2023-01-11 22:28:05.218307]
2023-01-11T22:28:05.2381397Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_pg_wrapper.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:28:05.218579]
2023-01-11T22:29:42.8235945Z 
2023-01-11T22:29:42.8236469Z Expand the folded group to see the log file of distributed/test_pg_wrapper
2023-01-11T22:29:42.8240285Z ##[group]PRINTING LOG FILE of distributed/test_pg_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-test_pg_wrapper_zx7kxg9i)
2023-01-11T22:29:42.8240803Z <unittest.suite.TestSuite tests=[]>
2023-01-11T22:29:42.8241126Z <unittest.suite.TestSuite tests=[]>
2023-01-11T22:29:42.8242839Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_hang>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_cuda_debug_mode>, <__main__.ProcessGroupGlooWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]>
2023-01-11T22:29:42.8244565Z test_collective_hang (__main__.ProcessGroupGlooWrapperTest)
2023-01-11T22:29:42.8245025Z test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest)
2023-01-11T22:29:42.8245509Z test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest)
2023-01-11T22:29:42.8245992Z test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2023-01-11T22:29:42.8246511Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2023-01-11T22:29:42.8247000Z test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest)
2023-01-11T22:29:42.8247444Z test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest)
2023-01-11T22:29:42.8247934Z test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2023-01-11T22:29:42.8248824Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest)
2023-01-11T22:29:42.8250823Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_hang>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collective_shape_mismatch_debug_mode>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch>, <__main__.ProcessGroupNCCLWrapperTest testMethod=test_collectives_op_mismatch_debug_mode>]>
2023-01-11T22:29:42.8251794Z test_collective_hang (__main__.ProcessGroupNCCLWrapperTest)
2023-01-11T22:29:42.8252393Z test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest)
2023-01-11T22:29:42.8253435Z test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest)
2023-01-11T22:29:42.8253938Z test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest)
2023-01-11T22:29:42.8254381Z test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest)
2023-01-11T22:29:42.8255113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8255760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8256351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8256822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8257040Z 
2023-01-11T22:29:42.8257149Z Running tests...
2023-01-11T22:29:42.8257574Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8258109Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T22:29:42.8258608Z   test_collective_hang (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:29:42.8259109Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65499
2023-01-11T22:29:42.8259556Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65500
2023-01-11T22:29:42.8260005Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65501
2023-01-11T22:29:42.8260422Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65502
2023-01-11T22:29:42.8261035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8261653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8262383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8262863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8263440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8263893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8264436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8264914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8265490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8265935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8266487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8266952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8267529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8267947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8268511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8268966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8269399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:42.8269850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:29:42.8270307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:29:42.8270774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:42.8271258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:42.8271728Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:42.8272899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:29:42.8273390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:29:42.8274047Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:29:42.8274730Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:29:42.8275407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:29:42.8276096Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:29:42.8276593Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2023-01-11T22:29:42.8277053Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 2000 ms
2023-01-11T22:29:42.8277633Z [E ProcessGroupGloo.cpp:138] Rank 2 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2023-01-11T22:29:42.8278308Z [E ProcessGroupGloo.cpp:138] Rank 3 successfully reached monitoredBarrier, but received errors while waiting for send/recv from rank 0. Please check rank 0 logs for faulty rank.
2023-01-11T22:29:42.8278736Z ok (4.290s)
2023-01-11T22:29:42.8278883Z 
2023-01-11T22:29:42.8279152Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8279612Z Ran 1 test in 4.291s
2023-01-11T22:29:42.8279834Z 
2023-01-11T22:29:42.8279934Z OK
2023-01-11T22:29:42.8280048Z 
2023-01-11T22:29:42.8280172Z Generating XML reports...
2023-01-11T22:29:42.8280798Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222808.xml
2023-01-11T22:29:42.8281526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8282009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8282581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8283044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8283274Z 
2023-01-11T22:29:42.8283382Z Running tests...
2023-01-11T22:29:42.8283762Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8284296Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T22:29:42.8284823Z   test_collective_shape_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:29:42.8285313Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65706
2023-01-11T22:29:42.8285755Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65707
2023-01-11T22:29:42.8286182Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65708
2023-01-11T22:29:42.8286624Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65709
2023-01-11T22:29:42.8287206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8287645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8288221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8288673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8289245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8289683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8290248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8290692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8291257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8291688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8292252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8292703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8293270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8293706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8294250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8294706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8295135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:42.8295604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:29:42.8296051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:29:42.8296565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:42.8297107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:42.8297581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:42.8298069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:29:42.8298546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:29:42.8299198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:29:42.8299883Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:29:42.8300563Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:29:42.8301224Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:29:42.8301613Z ok (4.254s)
2023-01-11T22:29:42.8301761Z 
2023-01-11T22:29:42.8302029Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8302338Z Ran 1 test in 4.254s
2023-01-11T22:29:42.8302500Z 
2023-01-11T22:29:42.8302593Z OK
2023-01-11T22:29:42.8302725Z 
2023-01-11T22:29:42.8302849Z Generating XML reports...
2023-01-11T22:29:42.8303469Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222815.xml
2023-01-11T22:29:42.8304173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8304620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8305192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8305656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8305865Z 
2023-01-11T22:29:42.8305971Z Running tests...
2023-01-11T22:29:42.8306369Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8306893Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T22:29:42.8307409Z   test_collective_shape_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:29:42.8307915Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65913
2023-01-11T22:29:42.8308355Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65914
2023-01-11T22:29:42.8308792Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 65915
2023-01-11T22:29:42.8309208Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 65916
2023-01-11T22:29:42.8309810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8310254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8310803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8311269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8312509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8313008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8313566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8314028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8314772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8315203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8315772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8316229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8316795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8317212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8317768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8318219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8318651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:42.8319113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:29:42.8319575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:42.8320040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:29:42.8320412Z skip: Need at least 4 CUDA devices (3.957s)
2023-01-11T22:29:42.8320606Z 
2023-01-11T22:29:42.8320874Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8321196Z Ran 1 test in 3.958s
2023-01-11T22:29:42.8321357Z 
2023-01-11T22:29:42.8321466Z OK (skipped=1)
2023-01-11T22:29:42.8321601Z 
2023-01-11T22:29:42.8321728Z Generating XML reports...
2023-01-11T22:29:42.8322353Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222821.xml
2023-01-11T22:29:42.8323079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8323508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8324075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8324538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8324765Z 
2023-01-11T22:29:42.8324875Z Running tests...
2023-01-11T22:29:42.8325256Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8325782Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T22:29:42.8326340Z   test_collective_shape_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:29:42.8326853Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66084
2023-01-11T22:29:42.8327307Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66085
2023-01-11T22:29:42.8327746Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66086
2023-01-11T22:29:42.8328187Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66087
2023-01-11T22:29:42.8328778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8329223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8329794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8330243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8330813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8331252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8331955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8332411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8332990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8333431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8333999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8334447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8335021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8335457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8336013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8336476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8336911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:29:42.8337385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:42.8337834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:42.8338292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:29:42.8338683Z skip: Need at least 4 CUDA devices (4.012s)
2023-01-11T22:29:42.8338871Z 
2023-01-11T22:29:42.8339120Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8339445Z Ran 1 test in 4.012s
2023-01-11T22:29:42.8339604Z 
2023-01-11T22:29:42.8339714Z OK (skipped=1)
2023-01-11T22:29:42.8339872Z 
2023-01-11T22:29:42.8339998Z Generating XML reports...
2023-01-11T22:29:42.8340605Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222828.xml
2023-01-11T22:29:42.8341334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8341777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8342324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8342788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8343017Z 
2023-01-11T22:29:42.8343129Z Running tests...
2023-01-11T22:29:42.8343532Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8344044Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T22:29:42.8344601Z   test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:29:42.8345129Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66255
2023-01-11T22:29:42.8345581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66256
2023-01-11T22:29:42.8346002Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66257
2023-01-11T22:29:42.8346430Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66258
2023-01-11T22:29:42.8347034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8347464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8348038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8348501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8349193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8349626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8350193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8350655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8351210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8351650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8352866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8353333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8353897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8354338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8354905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8355363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8355780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:29:42.8356253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:42.8356719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:42.8357166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:29:42.8357652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:42.8358154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:29:42.8358643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:29:42.8359116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:42.8359772Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:29:42.8360466Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:29:42.8361152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:29:42.8361823Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:29:42.8362362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:29:42.8362855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:29:42.8363324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T22:29:42.8363807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T22:29:42.8364457Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T22:29:42.8365151Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T22:29:42.8365816Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T22:29:42.8366670Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T22:29:42.8367071Z ok (4.281s)
2023-01-11T22:29:42.8367221Z 
2023-01-11T22:29:42.8367489Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8367794Z Ran 1 test in 4.282s
2023-01-11T22:29:42.8367953Z 
2023-01-11T22:29:42.8368046Z OK
2023-01-11T22:29:42.8368177Z 
2023-01-11T22:29:42.8368300Z Generating XML reports...
2023-01-11T22:29:42.8368907Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222834.xml
2023-01-11T22:29:42.8369634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8370082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8370655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8371110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8371336Z 
2023-01-11T22:29:42.8371445Z Running tests...
2023-01-11T22:29:42.8371842Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8372351Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T22:29:42.8372886Z   test_collectives_op_mismatch (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:29:42.8373395Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66474
2023-01-11T22:29:42.8373841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66475
2023-01-11T22:29:42.8374266Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66476
2023-01-11T22:29:42.8374700Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66477
2023-01-11T22:29:42.8375309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8375739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8376310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8376774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8377349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8377771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8378338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8378797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8379371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8379800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8380360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8380820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8381366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8381807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8382423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8382882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8383298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:29:42.8383883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:29:42.8384358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:42.8384806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:42.8385298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:42.8385792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:29:42.8386280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:29:42.8386752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:42.8387410Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:29:42.8388107Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:29:42.8388791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:29:42.8389457Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:29:42.8389848Z ok (4.259s)
2023-01-11T22:29:42.8389995Z 
2023-01-11T22:29:42.8390263Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8390571Z Ran 1 test in 4.259s
2023-01-11T22:29:42.8390729Z 
2023-01-11T22:29:42.8390821Z OK
2023-01-11T22:29:42.8390953Z 
2023-01-11T22:29:42.8391075Z Generating XML reports...
2023-01-11T22:29:42.8391696Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222840.xml
2023-01-11T22:29:42.8393109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8393565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8394135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8394598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8394807Z 
2023-01-11T22:29:42.8394916Z Running tests...
2023-01-11T22:29:42.8395315Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8395842Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T22:29:42.8396364Z   test_collectives_op_mismatch_cuda (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:29:42.8396880Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66681
2023-01-11T22:29:42.8397335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66682
2023-01-11T22:29:42.8397786Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66683
2023-01-11T22:29:42.8398210Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66684
2023-01-11T22:29:42.8398807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8399253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8399804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8400270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8400847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8401283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8402009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8402483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8403060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8403482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8404043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8404504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8405072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8405491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8406068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8406525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8406964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:42.8407420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:42.8407881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:29:42.8408347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:29:42.8408720Z skip: Need at least 4 CUDA devices (4.105s)
2023-01-11T22:29:42.8408910Z 
2023-01-11T22:29:42.8409178Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8409503Z Ran 1 test in 4.105s
2023-01-11T22:29:42.8409666Z 
2023-01-11T22:29:42.8409777Z OK (skipped=1)
2023-01-11T22:29:42.8409911Z 
2023-01-11T22:29:42.8410170Z Generating XML reports...
2023-01-11T22:29:42.8410799Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222847.xml
2023-01-11T22:29:42.8411529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8411959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8412530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8412990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8413218Z 
2023-01-11T22:29:42.8413328Z Running tests...
2023-01-11T22:29:42.8413710Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8414237Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T22:29:42.8414796Z   test_collectives_op_mismatch_cuda_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:29:42.8430759Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66852
2023-01-11T22:29:42.8431240Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66853
2023-01-11T22:29:42.8431693Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 66854
2023-01-11T22:29:42.8432818Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 66855
2023-01-11T22:29:42.8433480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8433935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8434512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8435195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8435792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8436239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8436807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8437253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8437826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8438268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8438814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8439270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8439848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8440288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8440837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8441296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8441727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:29:42.8442176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:29:42.8442644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:42.8443098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:42.8443490Z skip: Need at least 4 CUDA devices (4.109s)
2023-01-11T22:29:42.8443683Z 
2023-01-11T22:29:42.8443941Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8444263Z Ran 1 test in 4.109s
2023-01-11T22:29:42.8444423Z 
2023-01-11T22:29:42.8444531Z OK (skipped=1)
2023-01-11T22:29:42.8444685Z 
2023-01-11T22:29:42.8444790Z Generating XML reports...
2023-01-11T22:29:42.8445415Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222853.xml
2023-01-11T22:29:42.8446140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8446585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8447139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8447595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8447825Z 
2023-01-11T22:29:42.8447936Z Running tests...
2023-01-11T22:29:42.8448334Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8448844Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T22:29:42.8449385Z   test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupGlooWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:29:42.8449909Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67023
2023-01-11T22:29:42.8450341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67024
2023-01-11T22:29:42.8450782Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 67025
2023-01-11T22:29:42.8451225Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 67026
2023-01-11T22:29:42.8451823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8452370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8452954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8453419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8453979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8454416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8454983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8455438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8455991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8456436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8456999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8457457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8458004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8458437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8458999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8459435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8459865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:29:42.8460336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:29:42.8460808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:42.8461257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:42.8461733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:42.8462230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:42.8462705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:29:42.8463195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:29:42.8463848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:29:42.8464536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:29:42.8465212Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:29:42.8465899Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:29:42.8466427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:29:42.8466917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:29:42.8467389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T22:29:42.8467877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T22:29:42.8468524Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T22:29:42.8469337Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T22:29:42.8470007Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T22:29:42.8470690Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T22:29:42.8471084Z ok (4.409s)
2023-01-11T22:29:42.8471232Z 
2023-01-11T22:29:42.8471478Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8471797Z Ran 1 test in 4.410s
2023-01-11T22:29:42.8472464Z 
2023-01-11T22:29:42.8472564Z OK
2023-01-11T22:29:42.8472700Z 
2023-01-11T22:29:42.8472823Z Generating XML reports...
2023-01-11T22:29:42.8473444Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222900.xml
2023-01-11T22:29:42.8474183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8474633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8475185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8475650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8475877Z 
2023-01-11T22:29:42.8475984Z Running tests...
2023-01-11T22:29:42.8476377Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8476883Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T22:29:42.8477404Z   test_collective_hang (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:29:42.8477899Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67242
2023-01-11T22:29:42.8478361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67243
2023-01-11T22:29:42.8478952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8479398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8479967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8480415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8480983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8481421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8482027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8482473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8482916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:42.8483386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:42.8483853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:42.8484346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:42.8485000Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:42.8485685Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:42.8486190Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2023-01-11T22:29:42.8486660Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 2000 ms
2023-01-11T22:29:42.8487179Z ok (3.991s)
2023-01-11T22:29:42.8487333Z 
2023-01-11T22:29:42.8487604Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8487913Z Ran 1 test in 3.991s
2023-01-11T22:29:42.8488073Z 
2023-01-11T22:29:42.8488167Z OK
2023-01-11T22:29:42.8488299Z 
2023-01-11T22:29:42.8488422Z Generating XML reports...
2023-01-11T22:29:42.8489027Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111222906.xml
2023-01-11T22:29:42.8489761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8490211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8490782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8491231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8491468Z 
2023-01-11T22:29:42.8491576Z Running tests...
2023-01-11T22:29:42.8491969Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8492476Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T22:29:42.8493012Z   test_collective_shape_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:29:42.8493522Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67355
2023-01-11T22:29:42.8493971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67356
2023-01-11T22:29:42.8494557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8494999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8495565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8496022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8496593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8497029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8497593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8498034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8498469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:42.8498931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:42.8499400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:42.8499904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:42.8500555Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:42.8501241Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:42.8501614Z ok (5.493s)
2023-01-11T22:29:42.8501761Z 
2023-01-11T22:29:42.8502024Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8502345Z Ran 1 test in 5.493s
2023-01-11T22:29:42.8502502Z 
2023-01-11T22:29:42.8502594Z OK
2023-01-11T22:29:42.8502707Z 
2023-01-11T22:29:42.8502829Z Generating XML reports...
2023-01-11T22:29:42.8503446Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111222913.xml
2023-01-11T22:29:42.8504225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8504710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8505284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8505750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8505976Z 
2023-01-11T22:29:42.8506084Z Running tests...
2023-01-11T22:29:42.8506464Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8506986Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T22:29:42.8507530Z   test_collective_shape_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:29:42.8508041Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67485
2023-01-11T22:29:42.8508489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67486
2023-01-11T22:29:42.8509095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8509539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8510097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8510553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8511122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8511560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8512638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8513105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8513547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:42.8513998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:42.8514482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:42.8514984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:42.8515648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:42.8516323Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:42.8516848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:29:42.8517336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:29:42.8517997Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:29:42.8518665Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:29:42.8519058Z ok (5.581s)
2023-01-11T22:29:42.8519203Z 
2023-01-11T22:29:42.8519471Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8519776Z Ran 1 test in 5.581s
2023-01-11T22:29:42.8519933Z 
2023-01-11T22:29:42.8520025Z OK
2023-01-11T22:29:42.8520156Z 
2023-01-11T22:29:42.8520281Z Generating XML reports...
2023-01-11T22:29:42.8520905Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111222920.xml
2023-01-11T22:29:42.8521621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8522246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8522827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8523277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8523504Z 
2023-01-11T22:29:42.8523611Z Running tests...
2023-01-11T22:29:42.8524010Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8524534Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T22:29:42.8525050Z   test_collectives_op_mismatch (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:29:42.8525557Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67625
2023-01-11T22:29:42.8526005Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67626
2023-01-11T22:29:42.8526593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8527040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8527613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8528078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8528635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8529076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8529642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8530100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8530510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:42.8530996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:42.8531481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:42.8531938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:42.8532590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:42.8533274Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:42.8533649Z ok (5.505s)
2023-01-11T22:29:42.8533776Z 
2023-01-11T22:29:42.8534039Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8534357Z Ran 1 test in 5.505s
2023-01-11T22:29:42.8534512Z 
2023-01-11T22:29:42.8534597Z OK
2023-01-11T22:29:42.8534718Z 
2023-01-11T22:29:42.8534827Z Generating XML reports...
2023-01-11T22:29:42.8535427Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111222928.xml
2023-01-11T22:29:42.8536145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8536587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8537140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8537598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8537824Z 
2023-01-11T22:29:42.8537930Z Running tests...
2023-01-11T22:29:42.8538311Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8538892Z Test results will be stored in test-reports/python-unittest/distributed.test_pg_wrapper
2023-01-11T22:29:42.8539565Z   test_collectives_op_mismatch_debug_mode (__main__.ProcessGroupNCCLWrapperTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:29:42.8540095Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67755
2023-01-11T22:29:42.8540530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67756
2023-01-11T22:29:42.8541133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8541579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8542131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8542591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8543161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:42.8543604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:42.8544149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:42.8544611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:42.8545043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:42.8545512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:42.8545981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:42.8546473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:42.8547130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:42.8547806Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:42.8548345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:29:42.8548831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:29:42.8549475Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:29:42.8550137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:29:42.8550525Z ok (5.585s)
2023-01-11T22:29:42.8550672Z 
2023-01-11T22:29:42.8550935Z ----------------------------------------------------------------------
2023-01-11T22:29:42.8551258Z Ran 1 test in 5.585s
2023-01-11T22:29:42.8551400Z 
2023-01-11T22:29:42.8551492Z OK
2023-01-11T22:29:42.8551622Z 
2023-01-11T22:29:42.8551743Z Generating XML reports...
2023-01-11T22:29:42.8552868Z Generated XML report: test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111222936.xml
2023-01-11T22:29:42.8553249Z 
2023-01-11T22:29:42.8553576Z ##[endgroup]
2023-01-11T22:29:42.8554144Z FINISHED PRINTING LOG FILE of distributed/test_pg_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-test_pg_wrapper_zx7kxg9i)
2023-01-11T22:29:42.8554480Z 
2023-01-11T22:29:42.8554731Z Running distributed/fsdp/test_wrap ... [2023-01-11 22:29:42.824448]
2023-01-11T22:29:42.8555368Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_wrap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:29:42.824721]
2023-01-11T22:31:09.7504383Z 
2023-01-11T22:31:09.7505119Z Expand the folded group to see the log file of distributed/fsdp/test_wrap
2023-01-11T22:31:09.7506529Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_wrap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_wrap_znnhaoa_)
2023-01-11T22:31:09.7508164Z 
2023-01-11T22:31:09.7508770Z Running tests...
2023-01-11T22:31:09.7509736Z ----------------------------------------------------------------------
2023-01-11T22:31:09.7510796Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_wrap
2023-01-11T22:31:09.7511570Z   test_always_wrap (__main__.TestAutoWrap)
2023-01-11T22:31:09.7512656Z Test to ensure that if `always_wrap_policy` is ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:09.7514133Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:288: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2023-01-11T22:31:09.7515026Z   warnings.warn(
2023-01-11T22:31:09.7515452Z ok (1.671s)
2023-01-11T22:31:09.7518004Z   test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:31:09.7519706Z   warnings.warn(
2023-01-11T22:31:09.7519990Z ok (0.004s)
2023-01-11T22:31:09.7521106Z   test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7521804Z ok (0.004s)
2023-01-11T22:31:09.7522148Z   test_auto_wrap_api (__main__.TestAutoWrap)
2023-01-11T22:31:09.7522803Z Test to ensure with auto wrap, we wrap child modules correctly based on the min_num_params. ... ok (0.003s)
2023-01-11T22:31:09.7523490Z   test_auto_wrap_preset_exclude_wrap (__main__.TestAutoWrap)
2023-01-11T22:31:09.7523933Z Test to ensure excluded modules are not wrapped, regardless if the total param size is greater than the ... ok (0.002s)
2023-01-11T22:31:09.7524400Z   test_auto_wrap_preset_exclude_wrap_include_children (__main__.TestAutoWrap)
2023-01-11T22:31:09.7525581Z Test to ensure excluded modules are not wrapped, but children are if param size is greater than ... [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7526937Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7528175Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7528760Z ok (0.002s)
2023-01-11T22:31:09.7529041Z   test_auto_wrap_preset_force_leaf (__main__.TestAutoWrap)
2023-01-11T22:31:09.7529570Z Test to ensure force-leaf modules are not wrapped, and children are not wrapped. The ... ok (0.003s)
2023-01-11T22:31:09.7530001Z   test_auto_wrap_preset_force_leaf_custom (__main__.TestAutoWrap)
2023-01-11T22:31:09.7530438Z Test to ensure force-leaf modules are not wrapped. ... ok (0.002s)
2023-01-11T22:31:09.7531341Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7532205Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:31:09.7533244Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7534486Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7535062Z ok (0.466s)
2023-01-11T22:31:09.7535638Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7536483Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:31:09.7536887Z ok (0.042s)
2023-01-11T22:31:09.7537315Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... ok (0.002s)
2023-01-11T22:31:09.7537974Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... ok (0.002s)
2023-01-11T22:31:09.7538754Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7539601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:31:09.7540643Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7542130Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7542721Z ok (0.050s)
2023-01-11T22:31:09.7543299Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7544141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:31:09.7544531Z ok (0.042s)
2023-01-11T22:31:09.7545086Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7545928Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:31:09.7547079Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7548319Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7548914Z ok (0.052s)
2023-01-11T22:31:09.7549473Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7550325Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:31:09.7550717Z ok (0.043s)
2023-01-11T22:31:09.7551089Z   test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... ok (0.003s)
2023-01-11T22:31:09.7551571Z   test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... ok (0.003s)
2023-01-11T22:31:09.7552537Z   test_module_wrap_policy (__main__.TestAutoWrap)
2023-01-11T22:31:09.7553566Z Tests the ``ModuleWrapPolicy``. ... [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7554865Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7556102Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7557317Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7557906Z ok (0.027s)
2023-01-11T22:31:09.7558207Z   test_transformer_auto_wrap_policy (__main__.TestAutoWrap)
2023-01-11T22:31:09.7559229Z Tests the ``transformer_auto_wrap_policy``. ... [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7560506Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7561826Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7563101Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7564333Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7565555Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7566767Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7567980Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7569199Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7570415Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7571624Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7572830Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7573428Z ok (0.047s)
2023-01-11T22:31:09.7573749Z   test_wrap_disabled_outside_context (__main__.TestAutoWrap) ... ok (0.002s)
2023-01-11T22:31:09.7574817Z   test_wrap_override_defaults (__main__.TestAutoWrap) ... [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:31:09.7575469Z ok (0.002s)
2023-01-11T22:31:09.7575787Z   test_wrap_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... ok (0.002s)
2023-01-11T22:31:09.7576221Z   test_wrap_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... ok (0.002s)
2023-01-11T22:31:09.7576627Z   test_bn_always_wrapped_individually (__main__.TestFSDPWrap)
2023-01-11T22:31:09.7577184Z Ensures that by using _or_policy with _wrap_batchnorm_individually, even ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67927
2023-01-11T22:31:09.7577778Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67928
2023-01-11T22:31:09.7578386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7578835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7579389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7579857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7580437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7580876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7581428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7581890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7582342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7582822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:09.7583481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7584171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7584690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:09.7585143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:09.7586409Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:31:09.7587192Z   warnings.warn(
2023-01-11T22:31:09.7588387Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:31:09.7589167Z   warnings.warn(
2023-01-11T22:31:09.7589398Z dist init r=0, world=2
2023-01-11T22:31:09.7589646Z dist init r=1, world=2
2023-01-11T22:31:09.7589880Z ok (3.312s)
2023-01-11T22:31:09.7590234Z   test_error_already_wrapped_nested_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap)
2023-01-11T22:31:09.7590808Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68006
2023-01-11T22:31:09.7591342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68007
2023-01-11T22:31:09.7592424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7592888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7593469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7594099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7594669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7595109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7595677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7596138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7596575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:09.7597069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7597727Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7598420Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7598930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:09.7599397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:09.7600657Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:31:09.7601435Z   warnings.warn(
2023-01-11T22:31:09.7602589Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:31:09.7603348Z   warnings.warn(
2023-01-11T22:31:09.7603593Z dist init r=1, world=2
2023-01-11T22:31:09.7603838Z dist init r=0, world=2
2023-01-11T22:31:09.7604053Z ok (3.309s)
2023-01-11T22:31:09.7604428Z   test_error_already_wrapped_nested_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap)
2023-01-11T22:31:09.7604996Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68085
2023-01-11T22:31:09.7605532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68086
2023-01-11T22:31:09.7606128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7606578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7607145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7607594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7608167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7608606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7609171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7609616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7610068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7610673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:09.7611336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7612007Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7612528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:09.7612999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:09.7613334Z dist init r=0, world=2
2023-01-11T22:31:09.7613583Z dist init r=1, world=2
2023-01-11T22:31:09.7613818Z ok (3.309s)
2023-01-11T22:31:09.7614171Z   test_error_already_wrapped_nested_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap)
2023-01-11T22:31:09.7614744Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68164
2023-01-11T22:31:09.7615280Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68165
2023-01-11T22:31:09.7615885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7616315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7616883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7617344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7617915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7618338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7618905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7619374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7619808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:09.7620305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7620960Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7621647Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7622151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:09.7622618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:09.7623881Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:31:09.7624658Z   warnings.warn(
2023-01-11T22:31:09.7625808Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:31:09.7626660Z   warnings.warn(
2023-01-11T22:31:09.7626931Z dist init r=1, world=2
2023-01-11T22:31:09.7627185Z dist init r=0, world=2
2023-01-11T22:31:09.7627418Z ok (3.309s)
2023-01-11T22:31:09.7627775Z   test_error_already_wrapped_nested_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap)
2023-01-11T22:31:09.7628345Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68243
2023-01-11T22:31:09.7628878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68244
2023-01-11T22:31:09.7629469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7629915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7630487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7630959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7631515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7632478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7633071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7633514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7633965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7634459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:09.7635115Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7635795Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7636319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:09.7636787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:09.7637141Z dist init r=0, world=2
2023-01-11T22:31:09.7637369Z dist init r=1, world=2
2023-01-11T22:31:09.7637603Z ok (3.310s)
2023-01-11T22:31:09.7638233Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68322
2023-01-11T22:31:09.7638934Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68323
2023-01-11T22:31:09.7639541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7639993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7640564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7641016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7641591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7642031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7642582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7643045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7643493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:09.7644144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7644794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7645481Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7646003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:09.7646474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:09.7647724Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:31:09.7648503Z   warnings.warn(
2023-01-11T22:31:09.7649640Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:31:09.7650396Z   warnings.warn(
2023-01-11T22:31:09.7650640Z dist init r=1, world=2
2023-01-11T22:31:09.7650868Z dist init r=0, world=2
2023-01-11T22:31:09.7651102Z ok (3.812s)
2023-01-11T22:31:09.7651737Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68405
2023-01-11T22:31:09.7652459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68406
2023-01-11T22:31:09.7653047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7653493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7654058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7654505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7655073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7655510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7656076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7656525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7656976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:09.7657472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7658111Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7658799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7659316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:09.7659786Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:09.7660184Z dist init r=1, world=2
2023-01-11T22:31:09.7660476Z dist init r=0, world=2
2023-01-11T22:31:09.7660715Z ok (3.813s)
2023-01-11T22:31:09.7661330Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68488
2023-01-11T22:31:09.7662039Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68489
2023-01-11T22:31:09.7662645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7663088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7663640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7664106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7664682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7665125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7665671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7666133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7666583Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7667058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:09.7667711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7668396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7668924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:09.7669374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:09.7670642Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:31:09.7671412Z   warnings.warn(
2023-01-11T22:31:09.7673092Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:31:09.7673875Z   warnings.warn(
2023-01-11T22:31:09.7674101Z dist init r=0, world=2
2023-01-11T22:31:09.7674348Z dist init r=1, world=2
2023-01-11T22:31:09.7674580Z ok (3.812s)
2023-01-11T22:31:09.7675208Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68571
2023-01-11T22:31:09.7675906Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68572
2023-01-11T22:31:09.7676519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7677119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7677699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7678146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7678721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7679160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7679708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7680168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7680618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7681118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:09.7681760Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7682444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7682964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:09.7683434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:09.7683768Z dist init r=1, world=2
2023-01-11T22:31:09.7684015Z dist init r=0, world=2
2023-01-11T22:31:09.7684250Z ok (3.812s)
2023-01-11T22:31:09.7684863Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68654
2023-01-11T22:31:09.7685584Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68655
2023-01-11T22:31:09.7686189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7686633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7687184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7687651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7688221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7688697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7689265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7689733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7690184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7690660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:09.7691316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7692009Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7692530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:09.7692982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:09.7694301Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:31:09.7695129Z   warnings.warn(
2023-01-11T22:31:09.7696285Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:31:09.7697053Z   warnings.warn(
2023-01-11T22:31:09.7697280Z dist init r=1, world=2
2023-01-11T22:31:09.7697529Z dist init r=0, world=2
2023-01-11T22:31:09.7697766Z ok (3.812s)
2023-01-11T22:31:09.7698381Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68737
2023-01-11T22:31:09.7699094Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68738
2023-01-11T22:31:09.7699698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7700144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7700697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7701158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7701739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7702180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7702728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7703185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7703635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7704112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:09.7704765Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7705453Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7705980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:09.7706434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:09.7706782Z dist init r=0, world=2
2023-01-11T22:31:09.7707029Z dist init r=1, world=2
2023-01-11T22:31:09.7707248Z ok (3.811s)
2023-01-11T22:31:09.7707878Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68820
2023-01-11T22:31:09.7708590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68821
2023-01-11T22:31:09.7709193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7709617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7710293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7710758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7711335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7711758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7712914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7713376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7713809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:09.7714302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7714966Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7715653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7716158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:09.7716628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:09.7717888Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:31:09.7718668Z   warnings.warn(
2023-01-11T22:31:09.7719809Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:31:09.7720559Z   warnings.warn(
2023-01-11T22:31:09.7720802Z dist init r=1, world=2
2023-01-11T22:31:09.7721050Z dist init r=0, world=2
2023-01-11T22:31:09.7721265Z ok (3.812s)
2023-01-11T22:31:09.7721893Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68903
2023-01-11T22:31:09.7722614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68904
2023-01-11T22:31:09.7723222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7723651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7724222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7724687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7725257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7725679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7726241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7726855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7727315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:09.7727795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7728453Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7729144Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7729650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:09.7730119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:09.7730466Z dist init r=1, world=2
2023-01-11T22:31:09.7730713Z dist init r=0, world=2
2023-01-11T22:31:09.7730940Z ok (3.812s)
2023-01-11T22:31:09.7731573Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68986
2023-01-11T22:31:09.7732283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68987
2023-01-11T22:31:09.7732870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7733315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7733883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7734345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7734902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7735349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7735916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7736374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7736807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7737299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:09.7737951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7738620Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7739137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:09.7739617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:09.7739969Z dist init r=0, world=2
2023-01-11T22:31:09.7740199Z dist init r=1, world=2
2023-01-11T22:31:09.7740434Z ok (3.211s)
2023-01-11T22:31:09.7741065Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69065
2023-01-11T22:31:09.7741779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69066
2023-01-11T22:31:09.7742367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7742813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7743498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7743954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7744531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7744969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7745534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7745977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7746425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:09.7746918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7747558Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7748250Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7748772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:09.7749239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:09.7749576Z dist init r=1, world=2
2023-01-11T22:31:09.7749827Z dist init r=0, world=2
2023-01-11T22:31:09.7750061Z ok (3.812s)
2023-01-11T22:31:09.7750674Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69148
2023-01-11T22:31:09.7751389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69149
2023-01-11T22:31:09.7752523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7752986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7753549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7754015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7754584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7755024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7755573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7756033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7756489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7756969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:09.7757628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7758315Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7758832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:09.7759282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:09.7759637Z dist init r=0, world=2
2023-01-11T22:31:09.7759883Z dist init r=1, world=2
2023-01-11T22:31:09.7760100Z ok (3.311s)
2023-01-11T22:31:09.7760826Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69227
2023-01-11T22:31:09.7761610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69228
2023-01-11T22:31:09.7762219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7762647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7763216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7763682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7764252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7764674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7765246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7765704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7766153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:09.7766633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7767284Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7767971Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7768472Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:09.7768936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:09.7769291Z dist init r=0, world=2
2023-01-11T22:31:09.7769537Z dist init r=1, world=2
2023-01-11T22:31:09.7769752Z ok (3.813s)
2023-01-11T22:31:09.7770377Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69310
2023-01-11T22:31:09.7771088Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69311
2023-01-11T22:31:09.7771675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7772118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7772685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7773152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7773705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7774147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7774707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7775165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7775598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7776093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:09.7776743Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7777518Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7778044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:09.7778509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:09.7778863Z dist init r=0, world=2
2023-01-11T22:31:09.7779096Z dist init r=1, world=2
2023-01-11T22:31:09.7779331Z ok (3.210s)
2023-01-11T22:31:09.7779964Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69389
2023-01-11T22:31:09.7780674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69390
2023-01-11T22:31:09.7781264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7781719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7782287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7782731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7783303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7783745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7784308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7784750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7785200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:09.7785703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7786341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7787032Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7787552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:09.7788020Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:09.7788400Z dist init r=1, world=2
2023-01-11T22:31:09.7788651Z dist init r=0, world=2
2023-01-11T22:31:09.7788885Z ok (3.912s)
2023-01-11T22:31:09.7789498Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69472
2023-01-11T22:31:09.7790211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69473
2023-01-11T22:31:09.7790818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7791264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7791814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7792805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7793395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7793834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7794380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7794999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7795458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:09.7795951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7796593Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7797282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7797803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:09.7798253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:09.7798606Z dist init r=1, world=2
2023-01-11T22:31:09.7798860Z dist init r=0, world=2
2023-01-11T22:31:09.7799081Z ok (3.311s)
2023-01-11T22:31:09.7799710Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69551
2023-01-11T22:31:09.7800422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69552
2023-01-11T22:31:09.7801027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7801455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7802020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7802483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7803064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7803488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7804054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7804513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7804964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:09.7805442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7806096Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7806780Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7807343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:09.7807815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:09.7808166Z dist init r=0, world=2
2023-01-11T22:31:09.7808413Z dist init r=1, world=2
2023-01-11T22:31:09.7808632Z ok (3.812s)
2023-01-11T22:31:09.7809089Z   test_wrap_batchnorm_individually_use_or_policy_False (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69634
2023-01-11T22:31:09.7809636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69635
2023-01-11T22:31:09.7810230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7810677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7811245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7811819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7812385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7812829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7813395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7813835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7814285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:09.7814776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7815429Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7816109Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7816632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:09.7817100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:09.7818364Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:31:09.7819127Z   warnings.warn(
2023-01-11T22:31:09.7820267Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:31:09.7821043Z   warnings.warn(
2023-01-11T22:31:09.7821290Z dist init r=1, world=2
2023-01-11T22:31:09.7821534Z dist init r=0, world=2
2023-01-11T22:31:09.7821749Z ok (3.309s)
2023-01-11T22:31:09.7822201Z   test_wrap_batchnorm_individually_use_or_policy_True (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69713
2023-01-11T22:31:09.7822747Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69714
2023-01-11T22:31:09.7823336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7823793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7824363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7824827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7825384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:09.7825825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:09.7826385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:09.7826828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:09.7827275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:09.7827874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:09.7828533Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7829202Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:09.7829722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:09.7830191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:09.7831455Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:31:09.7832902Z   warnings.warn(
2023-01-11T22:31:09.7834057Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:31:09.7834836Z   warnings.warn(
2023-01-11T22:31:09.7835083Z dist init r=0, world=2
2023-01-11T22:31:09.7835330Z dist init r=1, world=2
2023-01-11T22:31:09.7835545Z ok (3.309s)
2023-01-11T22:31:09.7835694Z 
2023-01-11T22:31:09.7835966Z ----------------------------------------------------------------------
2023-01-11T22:31:09.7836291Z Ran 47 tests in 84.537s
2023-01-11T22:31:09.7836440Z 
2023-01-11T22:31:09.7836531Z OK
2023-01-11T22:31:09.7836665Z 
2023-01-11T22:31:09.7836789Z Generating XML reports...
2023-01-11T22:31:09.7837350Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20230111222944.xml
2023-01-11T22:31:09.7838040Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20230111222944.xml
2023-01-11T22:31:09.7838368Z 
2023-01-11T22:31:09.7838728Z ##[endgroup]
2023-01-11T22:31:09.7839288Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_wrap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_wrap_znnhaoa_)
2023-01-11T22:31:09.7839624Z 
2023-01-11T22:31:09.7839927Z Running distributed/_shard/sharded_tensor/test_sharded_tensor ... [2023-01-11 22:31:09.751387]
2023-01-11T22:31:09.7840645Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:31:09.751734]
2023-01-11T22:33:39.8257955Z 
2023-01-11T22:33:39.8260849Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/test_sharded_tensor
2023-01-11T22:33:39.8262135Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_9lbp4wsv)
2023-01-11T22:33:39.8262788Z 
2023-01-11T22:33:39.8262906Z Running tests...
2023-01-11T22:33:39.8263409Z ----------------------------------------------------------------------
2023-01-11T22:33:39.8264376Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor
2023-01-11T22:33:39.8265115Z   test_empty (__main__.TestCreateTensorFromParams) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:33:39.8265466Z ok (1.647s)
2023-01-11T22:33:39.8265886Z   test_local_tensor (__main__.TestLocalTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69827
2023-01-11T22:33:39.8269908Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69828
2023-01-11T22:33:39.8270600Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 69829
2023-01-11T22:33:39.8271138Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 69830
2023-01-11T22:33:39.8272301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8272786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8273640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8274120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8274978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8278079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8278836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8279579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8280201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8280913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8281489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8281969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8282553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8282982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8283571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8284043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8284671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8285159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8285690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8286376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8286772Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:33:39.8287328Z   test_local_tensor_error (__main__.TestLocalTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69963
2023-01-11T22:33:39.8288004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69964
2023-01-11T22:33:39.8288479Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 69965
2023-01-11T22:33:39.8289163Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 69966
2023-01-11T22:33:39.8289787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8290240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8290802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8291274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8291855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8292309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8293120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8293605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8294183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8294630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8295186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8295657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8296234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8296658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8297237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8297705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8298147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8298607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8299078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8299643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8300043Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8300503Z   test_collect_local_shard (__main__.TestModuleHookApi) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70099
2023-01-11T22:33:39.8301030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70100
2023-01-11T22:33:39.8301489Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 70101
2023-01-11T22:33:39.8301915Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 70102
2023-01-11T22:33:39.8302535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8302990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8303562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8304011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8304592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8305036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8305616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8306067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8306648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8307098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8307642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8308116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8308695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8309137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8310295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8310890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8311343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8312238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8312713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8313170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8313567Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8314023Z   test_reshard_output (__main__.TestModuleHookApi) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70235
2023-01-11T22:33:39.8314539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70236
2023-01-11T22:33:39.8314990Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 70237
2023-01-11T22:33:39.8315443Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 70238
2023-01-11T22:33:39.8316046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8316492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8317063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8317512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8318083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8318524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8319091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8319546Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8320122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8320562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8321131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8321575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8322150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8322585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8323137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8323597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8324039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8324509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8324955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8325409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8325798Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8326266Z   test_create_shard_with_no_placement (__main__.TestShardMetadata) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70371
2023-01-11T22:33:39.8326803Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70372
2023-01-11T22:33:39.8327250Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 70373
2023-01-11T22:33:39.8327844Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 70374
2023-01-11T22:33:39.8328445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8328889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8329461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8329931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8330486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8330929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8331499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8331942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8332524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8332962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8333523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8333963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8334534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8334973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8335521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8335982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8336428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8336905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8337356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8337812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8338201Z skip: Need at least 4 CUDA devices (2.309s)
2023-01-11T22:33:39.8338658Z   test_shard_metadata_init (__main__.TestShardMetadata) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70507
2023-01-11T22:33:39.8339183Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70508
2023-01-11T22:33:39.8339632Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 70509
2023-01-11T22:33:39.8340071Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 70510
2023-01-11T22:33:39.8340665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8341110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8341681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8342142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8342694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8343131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8343702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8344142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8344837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8345281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8345852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8346297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8346866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8347299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8347848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8348306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8348741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8349222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8349674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8350133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8350522Z skip: Need at least 4 CUDA devices (2.409s)
2023-01-11T22:33:39.8350978Z   test_shard_parameter (__main__.TestShardParameter) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70643
2023-01-11T22:33:39.8351504Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70644
2023-01-11T22:33:39.8352335Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 70645
2023-01-11T22:33:39.8352796Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 70646
2023-01-11T22:33:39.8353399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8353850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8354418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8354884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8355442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8355884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8356450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8356892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8357464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8357912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8358475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8358916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8359487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8359929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8360498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8360939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8361374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8361989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8362446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8362903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8363294Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8363780Z   test_shard_parameter_errors (__main__.TestShardParameter) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70779
2023-01-11T22:33:39.8364296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70780
2023-01-11T22:33:39.8364741Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 70781
2023-01-11T22:33:39.8365181Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 70782
2023-01-11T22:33:39.8365771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8366227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8366796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8367258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8367816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8368259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8368826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8369270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8369843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8370286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8370849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8371288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8371861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8372299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8372865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8373305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8373739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8374207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8374662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8375122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8375512Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:33:39.8375973Z   test_shard_tensor (__main__.TestShardTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70915
2023-01-11T22:33:39.8376464Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70916
2023-01-11T22:33:39.8376908Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 70917
2023-01-11T22:33:39.8377350Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 70918
2023-01-11T22:33:39.8377939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8378451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8379064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8379535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8380092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8380535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8381103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8381568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8382123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8382563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8383138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8383581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8384147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8384583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8385146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8385587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8386017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8386490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8386942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8387415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8387796Z skip: Need at least 4 CUDA devices (2.310s)
2023-01-11T22:33:39.8388258Z   test_shard_tensor_errors (__main__.TestShardTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71051
2023-01-11T22:33:39.8388756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71052
2023-01-11T22:33:39.8389201Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 71053
2023-01-11T22:33:39.8389645Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 71054
2023-01-11T22:33:39.8390230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8390678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8391252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8391716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8392713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8393305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8393959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8394487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8395094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8395599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8396373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8396932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8397588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8398097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8398776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8399255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8399819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8400362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8400904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8401435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8401920Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8420266Z   test_cleanup (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71187
2023-01-11T22:33:39.8420818Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71188
2023-01-11T22:33:39.8421256Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 71189
2023-01-11T22:33:39.8421716Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 71190
2023-01-11T22:33:39.8422363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8422815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8423365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8423833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8424406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8424878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8425448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8425913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8426489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8426911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8427476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8427943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8428517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8428939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8429504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8429963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8430381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8430855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8431313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8431785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8432787Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8433291Z   test_complete_world_size (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71323
2023-01-11T22:33:39.8433841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71324
2023-01-11T22:33:39.8434291Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 71325
2023-01-11T22:33:39.8434724Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 71326
2023-01-11T22:33:39.8435343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8435793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8436350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8436823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8437391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8437834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8438382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8438844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8439410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8439828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8440391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8440847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8441420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8441838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8442403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8442859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8443290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8443742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8444200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8444676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8445056Z skip: Need at least 4 CUDA devices (2.412s)
2023-01-11T22:33:39.8445428Z   test_create_sharded_tensor_like (__main__.TestShardedTensorChunked)
2023-01-11T22:33:39.8445965Z Test tensor like methods, i.e. torch.zeros_like(...), torch.full_like, etc. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71459
2023-01-11T22:33:39.8446505Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71460
2023-01-11T22:33:39.8446937Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 71461
2023-01-11T22:33:39.8447380Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 71462
2023-01-11T22:33:39.8447984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8448415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8448986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8449575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8450156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8450580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8451145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8451605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8452157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8452594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8453159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8453624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8454182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8454623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8455189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8455648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8456062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8456536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8457002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8457452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8457848Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:33:39.8458228Z   test_create_sharded_tensor_with_full (__main__.TestShardedTensorChunked)
2023-01-11T22:33:39.8458720Z Test sharded_tensor.full(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71595
2023-01-11T22:33:39.8459190Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71596
2023-01-11T22:33:39.8459637Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 71597
2023-01-11T22:33:39.8460088Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 71598
2023-01-11T22:33:39.8460677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8461123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8461696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8462170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8462726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8463174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8463730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8464153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8464718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8465182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8465763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8466306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8466885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8467328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8467893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8468334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8468769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8469240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8469686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8470155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8470550Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8470927Z   test_create_sharded_tensor_with_ones (__main__.TestShardedTensorChunked)
2023-01-11T22:33:39.8471397Z Test sharded_tensor.ones(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71731
2023-01-11T22:33:39.8472304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71732
2023-01-11T22:33:39.8472771Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 71733
2023-01-11T22:33:39.8473199Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 71734
2023-01-11T22:33:39.8473805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8474257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8474833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8475289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8475861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8476299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8476845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8477312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8477885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8478326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8478871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8479343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8479915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8480356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8480897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8481352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8481791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8482246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8482709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8483247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8483701Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8484063Z   test_create_sharded_tensor_with_rand (__main__.TestShardedTensorChunked)
2023-01-11T22:33:39.8484565Z Test sharded_tensor.rand(...)/randn(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71867
2023-01-11T22:33:39.8485063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71868
2023-01-11T22:33:39.8485494Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 71869
2023-01-11T22:33:39.8485948Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 71870
2023-01-11T22:33:39.8486553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8487001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8487554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8488022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8488600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8489024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8489585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8490046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8490617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8491037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8491601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8492068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8492642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8493064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8493627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8494084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8494499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8494971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8495434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8495913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8496289Z skip: Need at least 4 CUDA devices (2.310s)
2023-01-11T22:33:39.8496668Z   test_create_sharded_tensor_with_zeros (__main__.TestShardedTensorChunked)
2023-01-11T22:33:39.8497161Z Test sharded_tensor.zeros(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72003
2023-01-11T22:33:39.8497631Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72004
2023-01-11T22:33:39.8498080Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72005
2023-01-11T22:33:39.8498525Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72006
2023-01-11T22:33:39.8499129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8499602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8500229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8500756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8501312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8501756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8502325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8502784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8503339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8503782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8504347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8504812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8505367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8505807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8506366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8506804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8507238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8507708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8508175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8508634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8509020Z skip: Need at least 4 CUDA devices (2.310s)
2023-01-11T22:33:39.8509369Z   test_gather_even (__main__.TestShardedTensorChunked)
2023-01-11T22:33:39.8509862Z Test _sharded_tensor.gather(...) with evenly distributed._shards ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72139
2023-01-11T22:33:39.8510393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72140
2023-01-11T22:33:39.8510847Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72141
2023-01-11T22:33:39.8511296Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72142
2023-01-11T22:33:39.8512175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8512637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8513225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8513697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8514252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8514693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8515255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8515699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8516271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8516710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8517349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8517851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8518429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8518869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8519411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8519869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8520303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8520780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8521229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8521697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8522090Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8522427Z   test_gather_uneven (__main__.TestShardedTensorChunked)
2023-01-11T22:33:39.8522945Z Test _sharded_tensor.gather(...) with unevenly distributed._shards ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72275
2023-01-11T22:33:39.8523479Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72276
2023-01-11T22:33:39.8523926Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72277
2023-01-11T22:33:39.8524355Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72278
2023-01-11T22:33:39.8524962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8525412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8525990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8526437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8527010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8527453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8528005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8528461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8529032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8529472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8530024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8530483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8531047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8531465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8532024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8532485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8532919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8533371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8533832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8534402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8534778Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:33:39.8535284Z   test_insufficient_sharding_dims (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72411
2023-01-11T22:33:39.8535834Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72412
2023-01-11T22:33:39.8536284Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72413
2023-01-11T22:33:39.8536711Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72414
2023-01-11T22:33:39.8537314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8537756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8538328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8538784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8539354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8539789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8540333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8540791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8541359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8541793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8542339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8542800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8543374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8543795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8544354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8544809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8545242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8545695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8546151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8546625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8547016Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:33:39.8547490Z   test_invalid_pg_rpc_ranks (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72547
2023-01-11T22:33:39.8548034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72548
2023-01-11T22:33:39.8548474Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72549
2023-01-11T22:33:39.8548904Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72550
2023-01-11T22:33:39.8549507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8549953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8550507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8551033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8551604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8552367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8552932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8553373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8553938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8554398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8554962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8555436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8556008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8556456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8557005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8557467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8557903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8558354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8558816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8559285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8559685Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8560157Z   test_invalid_sharding (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72683
2023-01-11T22:33:39.8560694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72684
2023-01-11T22:33:39.8561143Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72685
2023-01-11T22:33:39.8561573Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72686
2023-01-11T22:33:39.8562179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8562625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8563194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8563649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8564222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8564658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8565202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8565660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8566227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8566665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8567207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8567664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8568376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8568820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8569365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8569820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8570252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8570703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8571170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8571631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8572022Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:33:39.8572502Z   test_load_state_dict_errors (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72819
2023-01-11T22:33:39.8573047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72820
2023-01-11T22:33:39.8573498Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72821
2023-01-11T22:33:39.8573926Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72822
2023-01-11T22:33:39.8574531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8574974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8575541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8575986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8576566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8577070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8577635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8578081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8578650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8579090Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8579634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8580090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8580668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8581104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8581652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8582112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8582545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8583001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8583466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8583926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8584312Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8584888Z   test_multiple_local_shards (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72955
2023-01-11T22:33:39.8585438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72956
2023-01-11T22:33:39.8585884Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 72957
2023-01-11T22:33:39.8586331Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 72958
2023-01-11T22:33:39.8586921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8587363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8587928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8588371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8588951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8589394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8589954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8590397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8590961Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8591396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8592159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8592624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8593207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8593648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8594193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8594653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8595085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8595559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8596004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8596465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8596855Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8597318Z   test_new_group (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73091
2023-01-11T22:33:39.8597851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73092
2023-01-11T22:33:39.8598298Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 73093
2023-01-11T22:33:39.8598745Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 73094
2023-01-11T22:33:39.8599333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8599817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8600390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8600841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8601415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8601996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8602574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8603021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8603594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8604033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8604576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8605034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8605603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8606047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8606592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8607052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8607481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8607955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8608403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8608866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8609253Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:33:39.8609729Z   test_partial_world_size (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73227
2023-01-11T22:33:39.8610279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73228
2023-01-11T22:33:39.8610728Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 73229
2023-01-11T22:33:39.8611172Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 73230
2023-01-11T22:33:39.8611759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8612202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8612776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8613224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8613796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8614233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8614782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8615222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8615787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8616213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8616772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8617205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8617765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8618202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8618869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8619337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8619773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8620244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8620688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8621151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8621543Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8622025Z   test_sharded_tensor_metadata (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73363
2023-01-11T22:33:39.8622579Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73364
2023-01-11T22:33:39.8623024Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 73365
2023-01-11T22:33:39.8623471Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 73366
2023-01-11T22:33:39.8624064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8624512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8625083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8625548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8626107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8626550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8627119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8627565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8628136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8628576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8629135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8629575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8630148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8630582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8631134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8631593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8632387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8632867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8633316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8633779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8634168Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:33:39.8634642Z   test_sharded_tensor_sizes (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73499
2023-01-11T22:33:39.8635181Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73500
2023-01-11T22:33:39.8635770Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 73501
2023-01-11T22:33:39.8636226Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 73502
2023-01-11T22:33:39.8636821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8637263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8637828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8638298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8638854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8639294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8639862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8640313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8640886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8641323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8641884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8642325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8642903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8643341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8643886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8644348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8644783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8645256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8645703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8646163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8646552Z skip: Need at least 4 CUDA devices (2.311s)
2023-01-11T22:33:39.8647043Z   test_sharding_columns (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73635
2023-01-11T22:33:39.8647561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73636
2023-01-11T22:33:39.8648017Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 73637
2023-01-11T22:33:39.8648464Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 73638
2023-01-11T22:33:39.8649052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8649497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8650072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8650540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8651095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8651535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8652098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8652674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8653255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8653689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8654259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8654698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8655268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8655705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8656270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8656717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8657148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8657622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8658068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8658533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8658925Z skip: Need at least 4 CUDA devices (2.310s)
2023-01-11T22:33:39.8659408Z   test_state_dict (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73771
2023-01-11T22:33:39.8659924Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73772
2023-01-11T22:33:39.8660374Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 73773
2023-01-11T22:33:39.8660832Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 73774
2023-01-11T22:33:39.8661419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8661867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8662433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8662895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8663451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8663892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8664456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8664905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8665474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8665917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8666483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8666925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8667493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8667932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8668495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8668935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8669463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8669941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8670390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8670853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8671241Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8671730Z   test_state_dict_new_group (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73907
2023-01-11T22:33:39.8672494Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73908
2023-01-11T22:33:39.8672947Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 73909
2023-01-11T22:33:39.8673405Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 73910
2023-01-11T22:33:39.8674005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8674453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8675023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8675486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8676043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8676484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8677050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8677514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8678074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8678515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8679075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8679521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8680086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8680525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8681089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8681530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8681969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8682439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8682890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8683355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8683740Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8684244Z   test_state_dict_no_sharded_tensors (__main__.TestShardedTensorChunked) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74043
2023-01-11T22:33:39.8684773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74044
2023-01-11T22:33:39.8685219Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 74045
2023-01-11T22:33:39.8685665Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 74046
2023-01-11T22:33:39.8686389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8686843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8687412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8687871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8688424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8688870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8689429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8689889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8690445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8690881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8691444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8691880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8692440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8692876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8693443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8693880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8694312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8694790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8695238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8695703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8696087Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8696571Z   test_custom_op (__main__.TestShardedTensorCustomOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74179
2023-01-11T22:33:39.8697089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74180
2023-01-11T22:33:39.8697540Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 74181
2023-01-11T22:33:39.8697987Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 74182
2023-01-11T22:33:39.8698595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8699027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8699597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8700111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8700670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8701111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8701676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8702133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8702687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8703228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8703798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8704240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8704816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8705257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8705815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8706246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8706683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8707166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8707638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8708090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8708478Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8708973Z   test_custom_op_errors (__main__.TestShardedTensorCustomOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74315
2023-01-11T22:33:39.8709503Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74316
2023-01-11T22:33:39.8709956Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 74317
2023-01-11T22:33:39.8710404Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 74318
2023-01-11T22:33:39.8711008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8711446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8712205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8712681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8713246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8713693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8714265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8714731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8715283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8715738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8716301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8716765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8717321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8717758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8718318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8718756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8719194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8719666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8720265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8720722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8721108Z skip: Need at least 4 CUDA devices (2.409s)
2023-01-11T22:33:39.8721604Z   test_custom_op_override (__main__.TestShardedTensorCustomOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74451
2023-01-11T22:33:39.8722137Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74452
2023-01-11T22:33:39.8722586Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 74453
2023-01-11T22:33:39.8723026Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 74454
2023-01-11T22:33:39.8723629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8724067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8724635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8725097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8725655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8726094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8726659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8727120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8727671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8728117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8728684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8729145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8729701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8730139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8730699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8731138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8731571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8732039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8732515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8732965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8733346Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8733732Z   test_create_sharded_tensor_with_ones (__main__.TestShardedTensorEnumerable)
2023-01-11T22:33:39.8734207Z Test sharded_tensor.ones(...) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74587
2023-01-11T22:33:39.8734700Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74588
2023-01-11T22:33:39.8735147Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 74589
2023-01-11T22:33:39.8735593Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 74590
2023-01-11T22:33:39.8736180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8736736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8737322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8737769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8738342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8738781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8739346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8739789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8740359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8740794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8741361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8741803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8742373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8742807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8743350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8743807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8744245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8744715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8745172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8745641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8746032Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8746370Z   test_gather_even (__main__.TestShardedTensorEnumerable)
2023-01-11T22:33:39.8746887Z Test _sharded_tensor.gather(...) with evenly distributed._shards ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74723
2023-01-11T22:33:39.8747422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74724
2023-01-11T22:33:39.8747870Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 74725
2023-01-11T22:33:39.8748300Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 74726
2023-01-11T22:33:39.8748902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8749357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8749913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8750383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8750959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8751397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8752117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8752591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8753170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8753735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8754294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8754753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8755322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8755741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8756300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8756752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8757183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8757640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8758107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8758573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8758946Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8759306Z   test_gather_uneven (__main__.TestShardedTensorEnumerable)
2023-01-11T22:33:39.8759831Z Test _sharded_tensor.gather(...) with unevenly distributed._shards ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74859
2023-01-11T22:33:39.8760370Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74860
2023-01-11T22:33:39.8760802Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 74861
2023-01-11T22:33:39.8761246Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 74862
2023-01-11T22:33:39.8761852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8762310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8762864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8763330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8763899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8764324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8764896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8765353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8765919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8766345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8766912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8767370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8767923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8768360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8768922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8769384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8769800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8770269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8770835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8771293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8771682Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8772176Z   test_grid_sharding (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74995
2023-01-11T22:33:39.8772719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74996
2023-01-11T22:33:39.8773149Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 74997
2023-01-11T22:33:39.8773595Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 74998
2023-01-11T22:33:39.8774199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8774654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8775194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8775638Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8776205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8776650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8777233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8777696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8778273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8778697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8779275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8779735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8780288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8780728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8781288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8781746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8782163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8782635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8783108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8783577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8783948Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:33:39.8784449Z   test_multiple_local_shards (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75131
2023-01-11T22:33:39.8785002Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75132
2023-01-11T22:33:39.8785435Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75133
2023-01-11T22:33:39.8785877Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75134
2023-01-11T22:33:39.8786484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8786929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8787588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8788057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8788632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8789051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8789616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8790075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8790641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8791059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8791635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8792267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8792848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8793268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8793829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8794289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8794704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8795178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8795644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8796122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8796494Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:33:39.8796983Z   test_new_group (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75267
2023-01-11T22:33:39.8797518Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75268
2023-01-11T22:33:39.8797949Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75269
2023-01-11T22:33:39.8798396Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75270
2023-01-11T22:33:39.8799000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8799443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8800047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8800508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8800873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8801047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8801423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8801611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8801952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8802125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8802567Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8802812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8803179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8803353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8803719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8803907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8804139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8804351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8804571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8804803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8804951Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:33:39.8805278Z   test_partial_world_size (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75403
2023-01-11T22:33:39.8805496Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75404
2023-01-11T22:33:39.8805712Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75405
2023-01-11T22:33:39.8805930Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75406
2023-01-11T22:33:39.8806283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8806459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8806836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8807034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8807398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8807571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8807943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8808134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8808497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8808649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8809018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8809212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8809577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8809749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8810117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8810303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8810532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8810743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8810968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8811193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8811440Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:33:39.8811782Z   test_sharded_tensor_device (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75539
2023-01-11T22:33:39.8812001Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75540
2023-01-11T22:33:39.8812218Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75541
2023-01-11T22:33:39.8812434Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75542
2023-01-11T22:33:39.8812805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8812962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8813339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8813536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8813896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8814070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8814440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8814628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8814985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8815138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8815505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8815690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8816062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8816234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8816599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8816785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8817015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8817242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8817446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8817668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8817825Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8818163Z   test_sharded_tensor_metadata (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75675
2023-01-11T22:33:39.8818382Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75676
2023-01-11T22:33:39.8818601Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75677
2023-01-11T22:33:39.8818817Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75678
2023-01-11T22:33:39.8819188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8819344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8819709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8819884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8820356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8820550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8820925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8821114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8821478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8821651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8822004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8822192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8822562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8822736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8823103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8823289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8823517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8823744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8823947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8824172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8824323Z skip: Need at least 4 CUDA devices (2.310s)
2023-01-11T22:33:39.8824667Z   test_sharded_tensor_to_cpu (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75811
2023-01-11T22:33:39.8824890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75812
2023-01-11T22:33:39.8825106Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75813
2023-01-11T22:33:39.8825320Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75814
2023-01-11T22:33:39.8825695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8825868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8826226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8826416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8826783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8826956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8827329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8827517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8827873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8828044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8828390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8828576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8828987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8829220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8829586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8829774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8830004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8830233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8830454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8830662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8830813Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:33:39.8831152Z   test_sharded_tensor_to_cuda (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75947
2023-01-11T22:33:39.8831371Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75948
2023-01-11T22:33:39.8831587Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 75949
2023-01-11T22:33:39.8831804Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 75950
2023-01-11T22:33:39.8832356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8832534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8832897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8833092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8833461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8833641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8834013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8834205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8834563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8834735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8835104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8835275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8835639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8835816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8836185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8836372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8836601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8836831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8837053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8837261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8837410Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:33:39.8837815Z   test_sharded_tensor_to_test (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76083
2023-01-11T22:33:39.8838092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76084
2023-01-11T22:33:39.8838309Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76085
2023-01-11T22:33:39.8838525Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76086
2023-01-11T22:33:39.8838894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8839070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8839448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8839618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8839980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8840160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8840526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8840699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8841075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8841265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8841641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8841812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8842176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8842355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8842725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8842912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8843141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8843370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8843591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8843819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8843949Z skip: Need at least 4 CUDA devices (2.412s)
2023-01-11T22:33:39.8844271Z   test_uneven_shards (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76219
2023-01-11T22:33:39.8844498Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76220
2023-01-11T22:33:39.8844714Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76221
2023-01-11T22:33:39.8844928Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76222
2023-01-11T22:33:39.8845297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8845470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8845843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8846015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8846376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8846606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8847021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8847212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8847577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8847750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8848119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8848305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8848647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8848817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8849191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8849376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8849606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8849831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8850056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8850282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8850412Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:33:39.8850736Z   test_with_rpc_names (__main__.TestShardedTensorEnumerable) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76355
2023-01-11T22:33:39.8850959Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76356
2023-01-11T22:33:39.8851176Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76357
2023-01-11T22:33:39.8851392Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76358
2023-01-11T22:33:39.8851758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8851931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8852305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8852494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8852835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8853008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8853388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8853578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8853935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8854109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8854478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8854663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8855003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8855173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8855590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8855822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8856054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8856281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8856504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8856733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8856880Z skip: Need at least 4 CUDA devices (2.311s)
2023-01-11T22:33:39.8857808Z   test_init_from_local_shards (__main__.TestShardedTensorFromLocalShards) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78068 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.002s)
2023-01-11T22:33:39.8858185Z   test_init_from_local_shards_and_global_metadata (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76491
2023-01-11T22:33:39.8858410Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76492
2023-01-11T22:33:39.8858626Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76493
2023-01-11T22:33:39.8858840Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76494
2023-01-11T22:33:39.8859210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8859383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8859758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8859953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8860295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8860470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8860841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8861019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8861391Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8861580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8861951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8862147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8862511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8862666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8863035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8863223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8863453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8863681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8863906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8864131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8864411Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:33:39.8864791Z   test_init_from_local_shards_and_global_metadata_invalid_shards (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76627
2023-01-11T22:33:39.8865012Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76628
2023-01-11T22:33:39.8865230Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76629
2023-01-11T22:33:39.8865444Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76630
2023-01-11T22:33:39.8865814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8865987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8866360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8866557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8866917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8867069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8867439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8867628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8867984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8868156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8868523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8868717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8869081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8869251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8869600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8869785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8870012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8870240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8870464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8870689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8870845Z skip: Need at least 4 CUDA devices (2.412s)
2023-01-11T22:33:39.8871221Z   test_init_from_local_shards_invalid_local_shards (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76763
2023-01-11T22:33:39.8871422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76764
2023-01-11T22:33:39.8871640Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76765
2023-01-11T22:33:39.8872079Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76766
2023-01-11T22:33:39.8872476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8872653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8873022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8873326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8873708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8873896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8874251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8874440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8874803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8874976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8875344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8875536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8875902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8876075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8876433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8876624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8876907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8877139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8877362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8877590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8877750Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:33:39.8878122Z   test_init_from_local_shards_invalid_pin_memory (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76899
2023-01-11T22:33:39.8878340Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76900
2023-01-11T22:33:39.8878540Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 76901
2023-01-11T22:33:39.8878755Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 76902
2023-01-11T22:33:39.8879125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8879298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8879674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8879872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8880238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8880410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8880762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8880949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8881305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8881475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8881842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8882026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8882489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8882669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8883035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8883203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8883432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8883658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8883878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8884103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8884357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:33:39.8884603Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:33:39.8884847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T22:33:39.8885251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:33:39.8885471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T22:33:39.8885874Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:33:39.8886269Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:33:39.8886662Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T22:33:39.8886813Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:33:39.8887197Z   test_init_from_local_shards_invalid_property_cross_ranks (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77047
2023-01-11T22:33:39.8887417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77048
2023-01-11T22:33:39.8887637Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 77049
2023-01-11T22:33:39.8887853Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 77050
2023-01-11T22:33:39.8888207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8888380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8888756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8888953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8889317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8889488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8889858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8890050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8890398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8890570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8890940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8891232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8891599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8891771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8892142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8892328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8892564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8892775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8892998Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8893223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8893379Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8893754Z   test_init_from_local_shards_invalid_shards_gaps (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77183
2023-01-11T22:33:39.8893974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77184
2023-01-11T22:33:39.8894191Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 77185
2023-01-11T22:33:39.8894405Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 77186
2023-01-11T22:33:39.8894757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8894932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8895295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8895475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8895848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8896036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8896411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8896600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8896962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8897120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8897493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8897685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8898052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8898224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8898593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8898777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8899006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8899215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8899439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8899704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8899953Z skip: Need at least 4 CUDA devices (2.309s)
2023-01-11T22:33:39.8900335Z   test_init_from_local_shards_invalid_shards_overlap (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77319
2023-01-11T22:33:39.8900554Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77320
2023-01-11T22:33:39.8900770Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 77321
2023-01-11T22:33:39.8900988Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 77322
2023-01-11T22:33:39.8901357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8901513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8901891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8902088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8902560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8902732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8903085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8903274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8903631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8903802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8904170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8904365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8904732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8904904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8905251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8905437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8905667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8905895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8906115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8906342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8906499Z skip: Need at least 4 CUDA devices (2.310s)
2023-01-11T22:33:39.8906859Z   test_init_from_local_shards_new_group (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77455
2023-01-11T22:33:39.8907078Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77456
2023-01-11T22:33:39.8907276Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 77457
2023-01-11T22:33:39.8907492Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 77458
2023-01-11T22:33:39.8907864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8908038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8908415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8908657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8909063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8909239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8909591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8909781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8910139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8910312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8910678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8910863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8911237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8911412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8911778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8912179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8912420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8912649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8912873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8913101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8913258Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:33:39.8913599Z   test_local_shards (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77591
2023-01-11T22:33:39.8913820Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77592
2023-01-11T22:33:39.8914019Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 77593
2023-01-11T22:33:39.8914235Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 77594
2023-01-11T22:33:39.8914615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8914789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8915165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8915352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8915719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8915892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8916265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8916438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8916797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8916967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8917331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8917517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8918016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8918192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8918559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8918728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8918957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8919183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8919406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8919631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8919782Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:33:39.8920171Z   test_st_base_init_from_local_shards_and_global_metadata (__main__.TestShardedTensorFromLocalShards) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77727
2023-01-11T22:33:39.8920389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77728
2023-01-11T22:33:39.8920605Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 77729
2023-01-11T22:33:39.8920805Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 77730
2023-01-11T22:33:39.8921175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8921347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8921718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8921907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8922274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8922446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8922815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8922984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8923343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8923512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8923880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8924064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8924431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8924604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8924970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8925153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8925364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8925594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8925815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8926041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8926189Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:33:39.8926583Z   test_init_from_local_tensor (__main__.TestShardedTensorFromLocalTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77863
2023-01-11T22:33:39.8926848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77864
2023-01-11T22:33:39.8927064Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 77865
2023-01-11T22:33:39.8927262Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 77866
2023-01-11T22:33:39.8927635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8927807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8928182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8928370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8928735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8928912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8929280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8929468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8929811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8929980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8930347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8930535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8930894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8931069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8931434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8931621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8931832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8932064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8932285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8932514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8932667Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8933024Z   test_init_from_local_tensor_errors (__main__.TestShardedTensorFromLocalTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77999
2023-01-11T22:33:39.8933247Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78000
2023-01-11T22:33:39.8933463Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 78001
2023-01-11T22:33:39.8933678Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 78002
2023-01-11T22:33:39.8934032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8934206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8934580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8934771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8935135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8935397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8935785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8935972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8936317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8936489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8936858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8937045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8937405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:39.8937586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:39.8937952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:39.8938138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:39.8938368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:39.8938576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:39.8938797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:33:39.8939023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:33:39.8939172Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:33:39.8939398Z   test_serialize_and_deserialize (__main__.TestShardedTensorMetadata) ... ok (0.051s)
2023-01-11T22:33:39.8939425Z 
2023-01-11T22:33:39.8939701Z ----------------------------------------------------------------------
2023-01-11T22:33:39.8939821Z Ran 64 tests in 147.739s
2023-01-11T22:33:39.8939840Z 
2023-01-11T22:33:39.8939950Z OK (skipped=62)
2023-01-11T22:33:39.8939969Z 
2023-01-11T22:33:39.8940076Z Generating XML reports...
2023-01-11T22:33:39.8940591Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestCreateTensorFromParams-20230111223111.xml
2023-01-11T22:33:39.8941095Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorMetadata-20230111223111.xml
2023-01-11T22:33:39.8941552Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestLocalTensor-20230111223111.xml
2023-01-11T22:33:39.8942019Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestModuleHookApi-20230111223111.xml
2023-01-11T22:33:39.8942481Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardMetadata-20230111223111.xml
2023-01-11T22:33:39.8942945Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardParameter-20230111223111.xml
2023-01-11T22:33:39.8943393Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardTensor-20230111223111.xml
2023-01-11T22:33:39.8943888Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorChunked-20230111223111.xml
2023-01-11T22:33:39.8944384Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorCustomOps-20230111223111.xml
2023-01-11T22:33:39.8944937Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorEnumerable-20230111223111.xml
2023-01-11T22:33:39.8945493Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalShards-20230111223111.xml
2023-01-11T22:33:39.8946017Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalTensor-20230111223111.xml
2023-01-11T22:33:39.8946037Z 
2023-01-11T22:33:39.8946547Z ##[endgroup]
2023-01-11T22:33:39.8947097Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_9lbp4wsv)
2023-01-11T22:33:39.8947117Z 
2023-01-11T22:33:39.8947389Z Running distributed/fsdp/test_fsdp_state_dict ... [2023-01-11 22:33:39.828266]
2023-01-11T22:33:39.8947879Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:33:39.828567]
2023-01-11T22:40:19.7298322Z 
2023-01-11T22:40:19.7302437Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_state_dict
2023-01-11T22:40:19.7303964Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_state_dict (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_state_dict_2vavclr5)
2023-01-11T22:40:19.7304608Z 
2023-01-11T22:40:19.7304787Z Running tests...
2023-01-11T22:40:19.7305583Z ----------------------------------------------------------------------
2023-01-11T22:40:19.7306589Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict
2023-01-11T22:40:19.7307731Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7308843Z Tests that we can save a state_dict and load it into a blank model ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:40:19.7309676Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78170
2023-01-11T22:40:19.7310459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78171
2023-01-11T22:40:19.7311515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7315332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7316393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7317238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7318275Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7319054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7320069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7323732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7324381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7325351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7326731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7327732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7328341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7328814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7331276Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7333700Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7335083Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7336348Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7336951Z dist init r=1, world=2
2023-01-11T22:40:19.7337184Z dist init r=0, world=2
2023-01-11T22:40:19.7337422Z ok (4.971s)
2023-01-11T22:40:19.7337942Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7338610Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78249
2023-01-11T22:40:19.7339289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78250
2023-01-11T22:40:19.7339938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7340389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7340962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7341414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7342181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7343080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7344170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7344972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7345903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7346962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7348314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7349692Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7350842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7351769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7352937Z dist init r=0, world=2
2023-01-11T22:40:19.7353417Z dist init r=1, world=2
2023-01-11T22:40:19.7353852Z ok (3.310s)
2023-01-11T22:40:19.7354928Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7356301Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78328
2023-01-11T22:40:19.7357331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78329
2023-01-11T22:40:19.7358466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7359364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7360474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7361319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7362427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7363233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7364475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7365505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7366525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7367519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7368841Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7370255Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7371368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7372269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7373123Z dist init r=0, world=2
2023-01-11T22:40:19.7392761Z dist init r=1, world=2
2023-01-11T22:40:19.7393013Z ok (3.410s)
2023-01-11T22:40:19.7393532Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7394222Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78407
2023-01-11T22:40:19.7394729Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78408
2023-01-11T22:40:19.7395903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7396706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7397729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7398548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7399524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7400278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7401274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7402058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7402866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7403725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7405253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7406187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7406706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7407168Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7407502Z dist init r=0, world=2
2023-01-11T22:40:19.7407749Z dist init r=1, world=2
2023-01-11T22:40:19.7407983Z ok (3.310s)
2023-01-11T22:40:19.7408473Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7409161Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78486
2023-01-11T22:40:19.7409687Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78487
2023-01-11T22:40:19.7410291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7410717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7411279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7411738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7412306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7412728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7414146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7414614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7415065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7415538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7416187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7416865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7417358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7417823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7418167Z dist init r=0, world=2
2023-01-11T22:40:19.7418417Z dist init r=1, world=2
2023-01-11T22:40:19.7418639Z ok (3.410s)
2023-01-11T22:40:19.7419147Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7419828Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78565
2023-01-11T22:40:19.7420328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78566
2023-01-11T22:40:19.7420931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7421379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7421945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7422509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7423093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7423534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7424093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7424533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7424978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7425473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7426102Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7426783Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7427296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7427759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7428092Z dist init r=1, world=2
2023-01-11T22:40:19.7428340Z dist init r=0, world=2
2023-01-11T22:40:19.7428577Z ok (3.310s)
2023-01-11T22:40:19.7429066Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7429805Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78644
2023-01-11T22:40:19.7430324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78645
2023-01-11T22:40:19.7430940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7431365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7432440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7432947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7433525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7433944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7434502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7434958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7435391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7435888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7436533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7437209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7437702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7438167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7438519Z dist init r=0, world=2
2023-01-11T22:40:19.7438748Z dist init r=1, world=2
2023-01-11T22:40:19.7438983Z ok (3.210s)
2023-01-11T22:40:19.7439587Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7440325Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78723
2023-01-11T22:40:19.7440833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78724
2023-01-11T22:40:19.7441440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7441886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7442451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7442900Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7443473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7443919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7444463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7444921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7445373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7445871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7446508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7447194Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7447712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7448189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7448526Z dist init r=1, world=2
2023-01-11T22:40:19.7448775Z dist init r=0, world=2
2023-01-11T22:40:19.7449011Z ok (3.310s)
2023-01-11T22:40:19.7449501Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7450178Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78802
2023-01-11T22:40:19.7450701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78803
2023-01-11T22:40:19.7451310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7451735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7452307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7452768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7453341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7453759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7454320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7454779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7455214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7455712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7456484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7457178Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7457682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7458151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7459148Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7460396Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7461641Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7462852Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7463455Z dist init r=0, world=2
2023-01-11T22:40:19.7463701Z dist init r=1, world=2
2023-01-11T22:40:19.7463942Z ok (3.310s)
2023-01-11T22:40:19.7464438Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7465125Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78881
2023-01-11T22:40:19.7465647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78882
2023-01-11T22:40:19.7466252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7466680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7467252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7467716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7468294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7468718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7469282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7469740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7470172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7470669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7471320Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7472631Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7473313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7473791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7474144Z dist init r=0, world=2
2023-01-11T22:40:19.7474376Z dist init r=1, world=2
2023-01-11T22:40:19.7474610Z ok (3.310s)
2023-01-11T22:40:19.7475121Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7475805Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78960
2023-01-11T22:40:19.7476315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78961
2023-01-11T22:40:19.7476932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7477381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7477950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7478399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7478968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7479405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7479953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7480413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7480865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7481369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7482005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7482696Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7483219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7483688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7484024Z dist init r=0, world=2
2023-01-11T22:40:19.7484269Z dist init r=1, world=2
2023-01-11T22:40:19.7484503Z ok (3.310s)
2023-01-11T22:40:19.7484988Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7485678Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79039
2023-01-11T22:40:19.7486203Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79040
2023-01-11T22:40:19.7486808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7487236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7487801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7488258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7488812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7489246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7489907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7490371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7490803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7491307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7491959Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7492649Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7493156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7493624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7493984Z dist init r=1, world=2
2023-01-11T22:40:19.7494213Z dist init r=0, world=2
2023-01-11T22:40:19.7494445Z ok (3.210s)
2023-01-11T22:40:19.7494952Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7495635Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79118
2023-01-11T22:40:19.7496141Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79119
2023-01-11T22:40:19.7496747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7497192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7497746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7498218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7498787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7499223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7499775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7500235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7500680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7501180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7501822Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7502510Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7503030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7503483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7503837Z dist init r=1, world=2
2023-01-11T22:40:19.7504085Z dist init r=0, world=2
2023-01-11T22:40:19.7504301Z ok (3.411s)
2023-01-11T22:40:19.7504803Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7505483Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79197
2023-01-11T22:40:19.7506106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79198
2023-01-11T22:40:19.7506703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7507144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7507711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7508173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7508727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7509168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7509726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7510192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7510626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7511124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7511779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7513033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7513556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7514024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7514373Z dist init r=1, world=2
2023-01-11T22:40:19.7514600Z dist init r=0, world=2
2023-01-11T22:40:19.7514831Z ok (3.210s)
2023-01-11T22:40:19.7515347Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7516010Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79276
2023-01-11T22:40:19.7516530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79277
2023-01-11T22:40:19.7517141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7517592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7518152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7518619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7519205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7519644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7520191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7520654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7521116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7521600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7522265Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7522949Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7523624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7524088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7524442Z dist init r=0, world=2
2023-01-11T22:40:19.7524695Z dist init r=1, world=2
2023-01-11T22:40:19.7524937Z ok (3.310s)
2023-01-11T22:40:19.7525426Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7526106Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79355
2023-01-11T22:40:19.7526630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79356
2023-01-11T22:40:19.7527222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7527678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7528291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7528757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7529384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7529807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7530382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7530852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7531283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7531808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7532484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7533175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7533699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7534154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7534504Z dist init r=1, world=2
2023-01-11T22:40:19.7534755Z dist init r=0, world=2
2023-01-11T22:40:19.7534979Z ok (3.311s)
2023-01-11T22:40:19.7535499Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7536202Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79434
2023-01-11T22:40:19.7536734Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79435
2023-01-11T22:40:19.7537326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7537777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7538346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7538795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7539371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7539820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7540485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7540934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7541409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7541907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7542569Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7543244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7543771Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7544243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7545266Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7546537Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7547743Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7548961Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7549657Z dist init r=0, world=2
2023-01-11T22:40:19.7549907Z dist init r=1, world=2
2023-01-11T22:40:19.7550141Z ok (3.410s)
2023-01-11T22:40:19.7550635Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7551327Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79513
2023-01-11T22:40:19.7552403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79514
2023-01-11T22:40:19.7553115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7553547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7554117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7554613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7555185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7555659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7556229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7556695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7557293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7557804Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7558469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7559162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7559669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7560143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7560500Z dist init r=0, world=2
2023-01-11T22:40:19.7560753Z dist init r=1, world=2
2023-01-11T22:40:19.7560973Z ok (3.410s)
2023-01-11T22:40:19.7561495Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7562191Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79592
2023-01-11T22:40:19.7562722Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79593
2023-01-11T22:40:19.7563371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7563858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7564437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7564888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7565466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7565938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7566549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7567000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7567464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7567973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7568636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7569307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7569831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7570312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7570649Z dist init r=0, world=2
2023-01-11T22:40:19.7570905Z dist init r=1, world=2
2023-01-11T22:40:19.7571144Z ok (3.310s)
2023-01-11T22:40:19.7571661Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7572333Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79671
2023-01-11T22:40:19.7572865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79672
2023-01-11T22:40:19.7573476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7574048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7574613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7575083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7575660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7576086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7576655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7577119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7577579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7578066Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7578730Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7579417Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7579947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7580399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7580749Z dist init r=1, world=2
2023-01-11T22:40:19.7581000Z dist init r=0, world=2
2023-01-11T22:40:19.7581220Z ok (3.310s)
2023-01-11T22:40:19.7581735Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7582428Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79750
2023-01-11T22:40:19.7582958Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79751
2023-01-11T22:40:19.7583549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7584006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7584579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7585029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7585605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7586049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7586625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7587075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7587535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7588040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7588700Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7589372Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7589897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7590375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7591461Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7593081Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7594326Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7595571Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7596179Z dist init r=0, world=2
2023-01-11T22:40:19.7596433Z dist init r=1, world=2
2023-01-11T22:40:19.7596654Z ok (3.411s)
2023-01-11T22:40:19.7597173Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7597865Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79829
2023-01-11T22:40:19.7598393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79830
2023-01-11T22:40:19.7598995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7599447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7600020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7600470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7601046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7601493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7602059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7602506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7602964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7603480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7604144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7604818Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7605343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7605815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7606155Z dist init r=0, world=2
2023-01-11T22:40:19.7606406Z dist init r=1, world=2
2023-01-11T22:40:19.7606645Z ok (3.311s)
2023-01-11T22:40:19.7607233Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7607996Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79908
2023-01-11T22:40:19.7608524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79909
2023-01-11T22:40:19.7609136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7609567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7610147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7610617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7611191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7611621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7612193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7612652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7613107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7613592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7614292Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7614983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7615490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7615973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7616329Z dist init r=0, world=2
2023-01-11T22:40:19.7616582Z dist init r=1, world=2
2023-01-11T22:40:19.7616803Z ok (3.210s)
2023-01-11T22:40:19.7617315Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7617995Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79987
2023-01-11T22:40:19.7618502Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79988
2023-01-11T22:40:19.7619114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7619564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7620146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7620599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7621177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7621620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7622188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7622634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7623085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7623588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7624385Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7625092Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7625616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7626092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7626429Z dist init r=0, world=2
2023-01-11T22:40:19.7626680Z dist init r=1, world=2
2023-01-11T22:40:19.7626920Z ok (3.310s)
2023-01-11T22:40:19.7627418Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7628111Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80066
2023-01-11T22:40:19.7628646Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80067
2023-01-11T22:40:19.7629308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7629745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7630321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7630791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7631368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7631795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7632788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7633277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7633717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7634225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7634886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7635581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7636089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7636564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7637562Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7638814Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7640047Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7641363Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7642014Z dist init r=0, world=2
2023-01-11T22:40:19.7642270Z dist init r=1, world=2
2023-01-11T22:40:19.7642513Z ok (3.410s)
2023-01-11T22:40:19.7643013Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7643711Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80145
2023-01-11T22:40:19.7644240Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80146
2023-01-11T22:40:19.7644851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7645296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7645874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7646346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7646926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7647351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7647919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7648388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7648829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7649342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7650001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7650694Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7651200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7651673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7652029Z dist init r=1, world=2
2023-01-11T22:40:19.7652261Z dist init r=0, world=2
2023-01-11T22:40:19.7652500Z ok (3.410s)
2023-01-11T22:40:19.7653018Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7653719Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80224
2023-01-11T22:40:19.7654225Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80225
2023-01-11T22:40:19.7654835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7655285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7655861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7656309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7656886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7657331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7657986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7658461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7658916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7659421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7660060Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7660747Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7661271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7661744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7662093Z dist init r=1, world=2
2023-01-11T22:40:19.7662347Z dist init r=0, world=2
2023-01-11T22:40:19.7662591Z ok (3.210s)
2023-01-11T22:40:19.7663083Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7663770Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80303
2023-01-11T22:40:19.7664295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80304
2023-01-11T22:40:19.7664907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7665340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7665919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7666391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7666950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7667396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7667962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7668423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7668858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7669361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7670020Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7670717Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7671225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7671702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7672444Z dist init r=0, world=2
2023-01-11T22:40:19.7672683Z dist init r=1, world=2
2023-01-11T22:40:19.7672924Z ok (3.310s)
2023-01-11T22:40:19.7673443Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7674135Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80382
2023-01-11T22:40:19.7674792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80383
2023-01-11T22:40:19.7675423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7675872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7676449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7676899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7677470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7677912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7678464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7678929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7679389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7679893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7680536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7681228Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7681755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7682233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7683215Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7684467Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7685710Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7686949Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7687563Z dist init r=0, world=2
2023-01-11T22:40:19.7687797Z dist init r=1, world=2
2023-01-11T22:40:19.7688035Z ok (3.411s)
2023-01-11T22:40:19.7688549Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7689238Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80461
2023-01-11T22:40:19.7689746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80462
2023-01-11T22:40:19.7690357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7690885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7691505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7691962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7692599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7693043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7693589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7694054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7694506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7695007Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7695659Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7696352Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7696878Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7697354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7697692Z dist init r=1, world=2
2023-01-11T22:40:19.7697942Z dist init r=0, world=2
2023-01-11T22:40:19.7698182Z ok (3.310s)
2023-01-11T22:40:19.7698683Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7699373Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80540
2023-01-11T22:40:19.7699911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80541
2023-01-11T22:40:19.7700522Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7700957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7701530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7701995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7702549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7702992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7703560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7704029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7704464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7704968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7705626Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7706314Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7706824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7707295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7707716Z dist init r=1, world=2
2023-01-11T22:40:19.7707991Z dist init r=0, world=2
2023-01-11T22:40:19.7708237Z ok (3.210s)
2023-01-11T22:40:19.7708750Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7709441Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80619
2023-01-11T22:40:19.7709950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80620
2023-01-11T22:40:19.7710560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7711014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7711566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7712424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7713024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7713468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7714015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7714482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7714940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7715449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7716092Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7716796Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7717321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7717776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7718135Z dist init r=0, world=2
2023-01-11T22:40:19.7718385Z dist init r=1, world=2
2023-01-11T22:40:19.7718605Z ok (3.311s)
2023-01-11T22:40:19.7719112Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7719790Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80698
2023-01-11T22:40:19.7720319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80699
2023-01-11T22:40:19.7720917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7721371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7721946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7722414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7722969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7723415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7723982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7724444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7725033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7725544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7726210Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7726882Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7727410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7727887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7728240Z dist init r=0, world=2
2023-01-11T22:40:19.7728470Z dist init r=1, world=2
2023-01-11T22:40:19.7728710Z ok (3.311s)
2023-01-11T22:40:19.7729224Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7729939Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80777
2023-01-11T22:40:19.7730470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80778
2023-01-11T22:40:19.7731082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7731534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7732088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7732555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7733130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7733581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7734133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7734594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7735053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7735538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7736202Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7736891Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7737412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7737871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7738226Z dist init r=1, world=2
2023-01-11T22:40:19.7738482Z dist init r=0, world=2
2023-01-11T22:40:19.7738703Z ok (3.410s)
2023-01-11T22:40:19.7739209Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7739887Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80856
2023-01-11T22:40:19.7740414Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80857
2023-01-11T22:40:19.7741008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7741458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7742140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7742618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7743180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7743628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7744199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7744645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7745102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7745609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7746276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7746943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7747475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7747956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7748955Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7750216Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7751446Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7753382Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7753994Z dist init r=0, world=2
2023-01-11T22:40:19.7754249Z dist init r=1, world=2
2023-01-11T22:40:19.7754499Z ok (3.311s)
2023-01-11T22:40:19.7754993Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7755679Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80935
2023-01-11T22:40:19.7756209Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80936
2023-01-11T22:40:19.7756823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7757262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7757838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7758310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7759028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7759489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7760063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7760531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7760965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7761477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7762134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7762827Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7763340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7763814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7764167Z dist init r=0, world=2
2023-01-11T22:40:19.7764397Z dist init r=1, world=2
2023-01-11T22:40:19.7764638Z ok (3.310s)
2023-01-11T22:40:19.7765139Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7765820Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81014
2023-01-11T22:40:19.7766328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81015
2023-01-11T22:40:19.7766951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7767402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7767972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7768425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7769002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7769453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7769995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7770458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7770916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7771426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7772076Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7772767Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7773292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7773769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7774107Z dist init r=1, world=2
2023-01-11T22:40:19.7774361Z dist init r=0, world=2
2023-01-11T22:40:19.7774599Z ok (3.410s)
2023-01-11T22:40:19.7775085Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7775876Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81093
2023-01-11T22:40:19.7776410Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81094
2023-01-11T22:40:19.7777023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7777453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7778030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7778501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7779059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7779504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7780083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7780550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7780986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7781495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7782157Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7782853Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7783355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7783829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7784195Z dist init r=0, world=2
2023-01-11T22:40:19.7784428Z dist init r=1, world=2
2023-01-11T22:40:19.7784667Z ok (3.410s)
2023-01-11T22:40:19.7785171Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7785851Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81172
2023-01-11T22:40:19.7786362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81173
2023-01-11T22:40:19.7786975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7787423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7787998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7788457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7789034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7789481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7790027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7790494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7790954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7791458Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7792345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7793195Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7793725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7794198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7795182Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7796437Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7797686Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7798293Z dist init r=0, world=2
2023-01-11T22:40:19.7798549Z dist init r=1, world=2
2023-01-11T22:40:19.7798765Z ok (3.411s)
2023-01-11T22:40:19.7799270Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7799950Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81251
2023-01-11T22:40:19.7800487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81252
2023-01-11T22:40:19.7801080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7801533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7802105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7802553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7803137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7803586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7804156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7804603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7805067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7805574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7806231Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7806905Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7807435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7807914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7808245Z dist init r=0, world=2
2023-01-11T22:40:19.7808497Z dist init r=1, world=2
2023-01-11T22:40:19.7808737Z ok (3.311s)
2023-01-11T22:40:19.7809359Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7810033Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81330
2023-01-11T22:40:19.7810560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81331
2023-01-11T22:40:19.7811172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7811624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7812180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7812649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7813237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7813664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7814236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7814702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7815157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7815643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7816299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7816991Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7817530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7817982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7818334Z dist init r=1, world=2
2023-01-11T22:40:19.7818584Z dist init r=0, world=2
2023-01-11T22:40:19.7818803Z ok (3.310s)
2023-01-11T22:40:19.7819311Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7819985Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81409
2023-01-11T22:40:19.7820511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81410
2023-01-11T22:40:19.7821101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7821556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7822127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7822595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7823156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7823598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7824157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7824605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7825058Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7825613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7826323Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7827000Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7827526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7827993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7828349Z dist init r=1, world=2
2023-01-11T22:40:19.7828576Z dist init r=0, world=2
2023-01-11T22:40:19.7828816Z ok (3.410s)
2023-01-11T22:40:19.7829458Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7830137Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81488
2023-01-11T22:40:19.7830672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81489
2023-01-11T22:40:19.7831279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7831729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7832618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7833090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7833663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7834094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7834672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7835142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7835598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7836078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7836740Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7837429Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7837950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7838409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7839413Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7840663Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7841921Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7843302Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7843911Z dist init r=0, world=2
2023-01-11T22:40:19.7844141Z dist init r=1, world=2
2023-01-11T22:40:19.7844382Z ok (3.310s)
2023-01-11T22:40:19.7844885Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7845552Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81567
2023-01-11T22:40:19.7846080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81568
2023-01-11T22:40:19.7846705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7847157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7847711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7848179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7848755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7849196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7849750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7850217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7850673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7851166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7851825Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7852520Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7853042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7853501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7853854Z dist init r=0, world=2
2023-01-11T22:40:19.7854101Z dist init r=1, world=2
2023-01-11T22:40:19.7854322Z ok (3.310s)
2023-01-11T22:40:19.7854831Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7855518Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81646
2023-01-11T22:40:19.7856040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81647
2023-01-11T22:40:19.7856636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7857094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7857664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7858134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7858687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7859235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7859803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7860273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7860709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7861213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7861866Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7862543Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7863065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7863544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7863892Z dist init r=0, world=2
2023-01-11T22:40:19.7864120Z dist init r=1, world=2
2023-01-11T22:40:19.7864356Z ok (3.411s)
2023-01-11T22:40:19.7864858Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7865520Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81725
2023-01-11T22:40:19.7866050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81726
2023-01-11T22:40:19.7866655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7867105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7867663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7868138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7868713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7869154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7869707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7870172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7870621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7871104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7871775Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7872723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7873252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7873704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7874063Z dist init r=1, world=2
2023-01-11T22:40:19.7874312Z dist init r=0, world=2
2023-01-11T22:40:19.7874529Z ok (3.410s)
2023-01-11T22:40:19.7875030Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7875705Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81804
2023-01-11T22:40:19.7876377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81805
2023-01-11T22:40:19.7876975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7877425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7877997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7878464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7879018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7879462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7880032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7880506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7880938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7881443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7882094Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7882791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7883293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7883764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7884758Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7886016Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.7886618Z dist init r=0, world=2
2023-01-11T22:40:19.7886868Z dist init r=1, world=2
2023-01-11T22:40:19.7887100Z ok (3.310s)
2023-01-11T22:40:19.7887598Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7888260Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81883
2023-01-11T22:40:19.7888792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81884
2023-01-11T22:40:19.7889400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7889842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7890396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7890861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7891436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7891866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7892526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7893035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7893490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7893976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7894640Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7895332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7895858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7896315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7896667Z dist init r=0, world=2
2023-01-11T22:40:19.7896921Z dist init r=1, world=2
2023-01-11T22:40:19.7897142Z ok (3.310s)
2023-01-11T22:40:19.7897617Z   test_fsdp_state_dict_keys_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81962
2023-01-11T22:40:19.7898213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81963
2023-01-11T22:40:19.7898823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7899257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7899828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7900295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7900851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7901304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7901870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7902328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7902763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7903262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7903918Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7904606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7905107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7905585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7906855Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.7907633Z   warnings.warn(
2023-01-11T22:40:19.7908812Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.7909643Z   warnings.warn(
2023-01-11T22:40:19.7909895Z dist init r=0, world=2
2023-01-11T22:40:19.7910143Z dist init r=1, world=2
2023-01-11T22:40:19.7910364Z ok (3.209s)
2023-01-11T22:40:19.7910835Z   test_fsdp_state_dict_keys_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82041
2023-01-11T22:40:19.7911401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82042
2023-01-11T22:40:19.7912237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7912678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7913263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7913739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7914298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7914738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7915309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7915766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7916202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7916704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7917363Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7918063Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7918568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7919039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7920311Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.7921098Z   warnings.warn(
2023-01-11T22:40:19.7922227Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.7922995Z   warnings.warn(
2023-01-11T22:40:19.7923244Z dist init r=1, world=2
2023-01-11T22:40:19.7923493Z dist init r=0, world=2
2023-01-11T22:40:19.7923711Z ok (3.309s)
2023-01-11T22:40:19.7924174Z   test_fsdp_state_dict_keys_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82120
2023-01-11T22:40:19.7924731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82121
2023-01-11T22:40:19.7925337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7925927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7926508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7926974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7927533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7927978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7928540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7928998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7929473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7929979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7930643Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7931335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7931841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7932315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7933580Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.7934365Z   warnings.warn(
2023-01-11T22:40:19.7935496Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.7936265Z   warnings.warn(
2023-01-11T22:40:19.7936511Z dist init r=0, world=2
2023-01-11T22:40:19.7936761Z dist init r=1, world=2
2023-01-11T22:40:19.7936979Z ok (3.309s)
2023-01-11T22:40:19.7937441Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7938242Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82199
2023-01-11T22:40:19.7938783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82200
2023-01-11T22:40:19.7939371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7939818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7940390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7940841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7941418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7941859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7942526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7942977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7943433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7943933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7944592Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7945267Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7945789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7946262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7947145Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.7947678Z   warnings.warn(
2023-01-11T22:40:19.7948438Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.7948984Z   warnings.warn(
2023-01-11T22:40:19.7949213Z dist init r=0, world=2
2023-01-11T22:40:19.7949461Z dist init r=1, world=2
2023-01-11T22:40:19.7949698Z ok (3.310s)
2023-01-11T22:40:19.7950163Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7950945Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82278
2023-01-11T22:40:19.7951486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82279
2023-01-11T22:40:19.7952333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7952770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7953354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7953823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7954395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7954821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7955399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7955861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7956316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7956801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7957461Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7958149Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7958655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7959129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7960150Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.7960711Z   warnings.warn(
2023-01-11T22:40:19.7961459Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.7962004Z   warnings.warn(
2023-01-11T22:40:19.7962254Z dist init r=0, world=2
2023-01-11T22:40:19.7962506Z dist init r=1, world=2
2023-01-11T22:40:19.7962724Z ok (3.410s)
2023-01-11T22:40:19.7963175Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7963957Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82357
2023-01-11T22:40:19.7964487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82358
2023-01-11T22:40:19.7965091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7965540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7966113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7966562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7967138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7967583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7968150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7968601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7969055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7969555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7970192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7970883Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7971406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7971879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7972737Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.7973287Z   warnings.warn(
2023-01-11T22:40:19.7974035Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.7974582Z   warnings.warn(
2023-01-11T22:40:19.7974810Z dist init r=1, world=2
2023-01-11T22:40:19.7975057Z dist init r=0, world=2
2023-01-11T22:40:19.7975297Z ok (3.310s)
2023-01-11T22:40:19.7975727Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7976557Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82436
2023-01-11T22:40:19.7977147Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82437
2023-01-11T22:40:19.7977756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7978187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7978759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7979223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7979793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7980212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7980782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7981248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7981680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7982182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7982839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7983532Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7984043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7984512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7985389Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.7985945Z   warnings.warn(
2023-01-11T22:40:19.7986685Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.7987225Z   warnings.warn(
2023-01-11T22:40:19.7987475Z dist init r=1, world=2
2023-01-11T22:40:19.7987706Z dist init r=0, world=2
2023-01-11T22:40:19.7987947Z ok (3.310s)
2023-01-11T22:40:19.7988401Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.7989185Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82515
2023-01-11T22:40:19.7989716Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82516
2023-01-11T22:40:19.7990314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7990760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7991310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7991775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7992634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.7993082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.7993633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.7994237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.7994693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.7995194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.7995841Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7996528Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.7997046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.7997499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.7998375Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.7998935Z   warnings.warn(
2023-01-11T22:40:19.7999692Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.8000234Z   warnings.warn(
2023-01-11T22:40:19.8000462Z dist init r=1, world=2
2023-01-11T22:40:19.8000714Z dist init r=0, world=2
2023-01-11T22:40:19.8000951Z ok (3.310s)
2023-01-11T22:40:19.8001383Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8002159Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82594
2023-01-11T22:40:19.8002711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82595
2023-01-11T22:40:19.8003295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8003745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8004317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8004780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8005334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8005778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8006347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8006815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8007252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8007760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8008421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8009098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8009620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8010093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8011011Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.8011590Z   warnings.warn(
2023-01-11T22:40:19.8012354Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.8012891Z   warnings.warn(
2023-01-11T22:40:19.8013138Z dist init r=0, world=2
2023-01-11T22:40:19.8013369Z dist init r=1, world=2
2023-01-11T22:40:19.8013605Z ok (3.310s)
2023-01-11T22:40:19.8014075Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8014856Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82673
2023-01-11T22:40:19.8015402Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82674
2023-01-11T22:40:19.8016005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8016452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8017007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8017472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8018048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8018485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8019031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8019496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8019950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8020435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8021091Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8021784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8022301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8022757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8023623Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.8024175Z   warnings.warn(
2023-01-11T22:40:19.8024924Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.8025447Z   warnings.warn(
2023-01-11T22:40:19.8025695Z dist init r=1, world=2
2023-01-11T22:40:19.8025942Z dist init r=0, world=2
2023-01-11T22:40:19.8026160Z ok (3.311s)
2023-01-11T22:40:19.8026629Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8027427Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82752
2023-01-11T22:40:19.8028066Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82753
2023-01-11T22:40:19.8028661Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8029111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8029730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8030199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8030753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8031197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8031763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8032453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8032916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8033416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8034081Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8034756Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8035277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8035748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8036623Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.8037163Z   warnings.warn(
2023-01-11T22:40:19.8037922Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.8038466Z   warnings.warn(
2023-01-11T22:40:19.8038706Z dist init r=0, world=2
2023-01-11T22:40:19.8038957Z dist init r=1, world=2
2023-01-11T22:40:19.8039194Z ok (3.310s)
2023-01-11T22:40:19.8039631Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8040413Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82831
2023-01-11T22:40:19.8040964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82832
2023-01-11T22:40:19.8041573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8042002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8042574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8043036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8043593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8044035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8044602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8045063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8045651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8046158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8046817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8047511Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8048015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8048485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8049356Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.8049912Z   warnings.warn(
2023-01-11T22:40:19.8050648Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.8051189Z   warnings.warn(
2023-01-11T22:40:19.8051435Z dist init r=1, world=2
2023-01-11T22:40:19.8051664Z dist init r=0, world=2
2023-01-11T22:40:19.8051900Z ok (3.310s)
2023-01-11T22:40:19.8052353Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8053140Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82910
2023-01-11T22:40:19.8053670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82911
2023-01-11T22:40:19.8054320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8054772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8055341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8055789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8056368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8056808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8057368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8057810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8058270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8058768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8059420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8060092Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8060609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8061077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8061946Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.8062579Z   warnings.warn(
2023-01-11T22:40:19.8063336Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.8063880Z   warnings.warn(
2023-01-11T22:40:19.8064109Z dist init r=0, world=2
2023-01-11T22:40:19.8064359Z dist init r=1, world=2
2023-01-11T22:40:19.8064597Z ok (3.310s)
2023-01-11T22:40:19.8065034Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8065815Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82989
2023-01-11T22:40:19.8066356Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82990
2023-01-11T22:40:19.8066968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8067397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8067965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8068430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8069001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8069421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8069985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8070442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8070701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8070949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8071335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8071731Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8072202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8072442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8072555Z dist init r=1, world=2
2023-01-11T22:40:19.8072664Z dist init r=0, world=2
2023-01-11T22:40:19.8072763Z ok (3.310s)
2023-01-11T22:40:19.8073101Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8073562Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83068
2023-01-11T22:40:19.8073765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83069
2023-01-11T22:40:19.8074135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8074309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8074686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8074873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8075231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8075539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8075923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8076091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8076339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8076584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8076984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8077379Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8077609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8077846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8077959Z dist init r=0, world=2
2023-01-11T22:40:19.8078067Z dist init r=1, world=2
2023-01-11T22:40:19.8078162Z ok (3.310s)
2023-01-11T22:40:19.8078484Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8078937Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83147
2023-01-11T22:40:19.8079156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83148
2023-01-11T22:40:19.8079523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8079696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8080079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8080274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8080622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8080796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8081168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8081354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8081599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8081841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8082240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8082643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8082873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8083085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8083195Z dist init r=0, world=2
2023-01-11T22:40:19.8083304Z dist init r=1, world=2
2023-01-11T22:40:19.8083404Z ok (3.310s)
2023-01-11T22:40:19.8083721Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8084168Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83226
2023-01-11T22:40:19.8084484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83227
2023-01-11T22:40:19.8084859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8085015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8085390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8085580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8085943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8086114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8086483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8086666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8086919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8087163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8087547Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8087943Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8088172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8088399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8088510Z dist init r=0, world=2
2023-01-11T22:40:19.8088617Z dist init r=1, world=2
2023-01-11T22:40:19.8088715Z ok (3.310s)
2023-01-11T22:40:19.8089036Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8089471Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83305
2023-01-11T22:40:19.8089690Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83306
2023-01-11T22:40:19.8090058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8090231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8090605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8090794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8091156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8091334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8091704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8091872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8092116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8092359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8092759Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8093152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8093380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8093699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8093815Z dist init r=0, world=2
2023-01-11T22:40:19.8093905Z dist init r=1, world=2
2023-01-11T22:40:19.8094004Z ok (3.310s)
2023-01-11T22:40:19.8094320Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8094773Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83384
2023-01-11T22:40:19.8094992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83385
2023-01-11T22:40:19.8095358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8095531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8095914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8096107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8096454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8096627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8096996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8097184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8097430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8097671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8098074Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8098470Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8098698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8098909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8099018Z dist init r=0, world=2
2023-01-11T22:40:19.8099127Z dist init r=1, world=2
2023-01-11T22:40:19.8099226Z ok (3.310s)
2023-01-11T22:40:19.8099564Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8100012Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83463
2023-01-11T22:40:19.8100235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83464
2023-01-11T22:40:19.8100606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8100761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8101138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8101326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8101689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8101861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8102229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8102506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8102759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8102982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8103382Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8103777Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8104002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8104228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8104339Z dist init r=0, world=2
2023-01-11T22:40:19.8104447Z dist init r=1, world=2
2023-01-11T22:40:19.8104549Z ok (3.310s)
2023-01-11T22:40:19.8104869Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8105322Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83542
2023-01-11T22:40:19.8105541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83543
2023-01-11T22:40:19.8105906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8106079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8106453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8106643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8107012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8107189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8107541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8107727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8107970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8108213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8108610Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8109004Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8109240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8109471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8109582Z dist init r=0, world=2
2023-01-11T22:40:19.8109671Z dist init r=1, world=2
2023-01-11T22:40:19.8109771Z ok (3.310s)
2023-01-11T22:40:19.8110091Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8110537Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83621
2023-01-11T22:40:19.8110756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83622
2023-01-11T22:40:19.8111123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8111351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8111815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8112223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8112607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8112780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8113148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8113333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8113577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8113821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8114232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8114629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8114838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8115066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8115178Z dist init r=1, world=2
2023-01-11T22:40:19.8115284Z dist init r=0, world=2
2023-01-11T22:40:19.8115387Z ok (3.311s)
2023-01-11T22:40:19.8115709Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8116160Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83700
2023-01-11T22:40:19.8116384Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83701
2023-01-11T22:40:19.8116733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8116908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8117285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8117473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8117838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8118007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8118378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8118571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8118816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8119043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8119440Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8119837Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8120062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8120288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8120398Z dist init r=0, world=2
2023-01-11T22:40:19.8120607Z dist init r=1, world=2
2023-01-11T22:40:19.8120706Z ok (3.310s)
2023-01-11T22:40:19.8121063Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8121387Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83779
2023-01-11T22:40:19.8121606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83780
2023-01-11T22:40:19.8121976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8122147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8122525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8122716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8123088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8123259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8123611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8123798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8124046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8124287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8124684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8125080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8125313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8125544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8125654Z dist init r=1, world=2
2023-01-11T22:40:19.8125745Z dist init r=0, world=2
2023-01-11T22:40:19.8125847Z ok (3.811s)
2023-01-11T22:40:19.8126171Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8126490Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83862
2023-01-11T22:40:19.8126706Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83863
2023-01-11T22:40:19.8127076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8127258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8127637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8127808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8128174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8128345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8128711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8128895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8129137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8129525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8129939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8130336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8130546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8130772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8130885Z dist init r=1, world=2
2023-01-11T22:40:19.8130990Z dist init r=0, world=2
2023-01-11T22:40:19.8131089Z ok (3.210s)
2023-01-11T22:40:19.8131410Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8131734Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83941
2023-01-11T22:40:19.8131951Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83942
2023-01-11T22:40:19.8132301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8132476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8132851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8133040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8133406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8133575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8133950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8134136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8134363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8134606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8135003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8135394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8135620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8135845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8135960Z dist init r=0, world=2
2023-01-11T22:40:19.8136071Z dist init r=1, world=2
2023-01-11T22:40:19.8136169Z ok (3.811s)
2023-01-11T22:40:19.8136473Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8136788Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84024
2023-01-11T22:40:19.8137001Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84025
2023-01-11T22:40:19.8137368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8137542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8137917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8138205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8138577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8138730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8139101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8139285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8139527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8139767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8140163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8140566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8140792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8141018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8141111Z dist init r=0, world=2
2023-01-11T22:40:19.8141219Z dist init r=1, world=2
2023-01-11T22:40:19.8141317Z ok (3.310s)
2023-01-11T22:40:19.8141643Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8141960Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84103
2023-01-11T22:40:19.8142180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84104
2023-01-11T22:40:19.8142554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8142725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8143087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8143277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8143638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8143810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8144178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8144361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8144606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8144855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8145253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8145629Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8145855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8146079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8146190Z dist init r=0, world=2
2023-01-11T22:40:19.8146297Z dist init r=1, world=2
2023-01-11T22:40:19.8146394Z ok (3.911s)
2023-01-11T22:40:19.8146721Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8147131Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84186
2023-01-11T22:40:19.8147357Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84187
2023-01-11T22:40:19.8147715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8147888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8148259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8148445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8148807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8148975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8149353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8149540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8149766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8150008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8150408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8150800Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8151029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8151255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8151370Z dist init r=0, world=2
2023-01-11T22:40:19.8151477Z dist init r=1, world=2
2023-01-11T22:40:19.8151558Z ok (3.310s)
2023-01-11T22:40:19.8152113Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8152445Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84265
2023-01-11T22:40:19.8152663Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84266
2023-01-11T22:40:19.8153042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8153216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8153598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8153790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8154155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8154308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8154682Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8154868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8155114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8155356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8155754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8156289Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8156526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8156750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8156843Z dist init r=0, world=2
2023-01-11T22:40:19.8156951Z dist init r=1, world=2
2023-01-11T22:40:19.8157046Z ok (3.811s)
2023-01-11T22:40:19.8157368Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8157679Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84348
2023-01-11T22:40:19.8157896Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84349
2023-01-11T22:40:19.8158277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8158448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8158810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8158999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8159365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8159535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8159905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8160092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8160345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8160585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8160983Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8161364Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8161591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8161821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8161928Z dist init r=0, world=2
2023-01-11T22:40:19.8162034Z dist init r=1, world=2
2023-01-11T22:40:19.8162132Z ok (3.210s)
2023-01-11T22:40:19.8162453Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8162771Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84427
2023-01-11T22:40:19.8162971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84428
2023-01-11T22:40:19.8163341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8163514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8163888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8164075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8164437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8164696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8165076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8165261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8165489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8165726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8166123Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8166519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8166745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8166973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8167081Z dist init r=1, world=2
2023-01-11T22:40:19.8167185Z dist init r=0, world=2
2023-01-11T22:40:19.8167265Z ok (3.811s)
2023-01-11T22:40:19.8167578Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8167894Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84510
2023-01-11T22:40:19.8168111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84511
2023-01-11T22:40:19.8168482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8168654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8169037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8169226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8169591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8169746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8170113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8170296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8170538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8170776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8171176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8171578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8171804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8172031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8172125Z dist init r=0, world=2
2023-01-11T22:40:19.8172230Z dist init r=1, world=2
2023-01-11T22:40:19.8172329Z ok (3.911s)
2023-01-11T22:40:19.8172644Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8172961Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84593
2023-01-11T22:40:19.8173276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84594
2023-01-11T22:40:19.8173655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8173825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8174185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8174373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8174736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8174908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8175277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8175466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8175711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8175951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8176331Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8176732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8176962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8177186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8177296Z dist init r=1, world=2
2023-01-11T22:40:19.8177402Z dist init r=0, world=2
2023-01-11T22:40:19.8177503Z ok (3.811s)
2023-01-11T22:40:19.8177823Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8178141Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84676
2023-01-11T22:40:19.8178343Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84677
2023-01-11T22:40:19.8178706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8178879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8179253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8179504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8200998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8201231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8201667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8201854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8202098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8202333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8202722Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8203112Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8203467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8203752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8203858Z dist init r=0, world=2
2023-01-11T22:40:19.8203959Z dist init r=1, world=2
2023-01-11T22:40:19.8204052Z ok (3.812s)
2023-01-11T22:40:19.8204361Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8204783Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84759
2023-01-11T22:40:19.8205000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84760
2023-01-11T22:40:19.8205363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8205528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8205909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8206095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8206455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8206614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8206982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8207151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8207394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8207637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8208045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8208440Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8208669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8208897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8209525Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.8209637Z   warnings.warn(
2023-01-11T22:40:19.8210246Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.8210359Z   warnings.warn(
2023-01-11T22:40:19.8210470Z dist init r=1, world=2
2023-01-11T22:40:19.8210577Z dist init r=0, world=2
2023-01-11T22:40:19.8210677Z ok (3.811s)
2023-01-11T22:40:19.8210987Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8211418Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84842
2023-01-11T22:40:19.8211638Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84843
2023-01-11T22:40:19.8211992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8212167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8212649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8212842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8213208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8213381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8213750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8213933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8214177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8214404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8214809Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8215212Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8215441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8215668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8216689Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.8216804Z   warnings.warn(
2023-01-11T22:40:19.8217810Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.8217920Z   warnings.warn(
2023-01-11T22:40:19.8218531Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.8218628Z   warnings.warn(
2023-01-11T22:40:19.8219239Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.8219335Z   warnings.warn(
2023-01-11T22:40:19.8219435Z dist init r=1, world=2
2023-01-11T22:40:19.8219540Z dist init r=0, world=2
2023-01-11T22:40:19.8219639Z ok (3.811s)
2023-01-11T22:40:19.8219949Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8220382Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84925
2023-01-11T22:40:19.8220601Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84926
2023-01-11T22:40:19.8220966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8221126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8221599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8221794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8222160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8222332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8222701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8222891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8223138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8223363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8223766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8224166Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8224394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8224621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8224731Z dist init r=1, world=2
2023-01-11T22:40:19.8224837Z dist init r=0, world=2
2023-01-11T22:40:19.8224936Z ok (3.210s)
2023-01-11T22:40:19.8225228Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8225657Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85004
2023-01-11T22:40:19.8225882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85005
2023-01-11T22:40:19.8226253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8226427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8226802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8226989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8227350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8227521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8227874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8228061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8228316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8228558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8228956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8229351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8229629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8229862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8229974Z dist init r=1, world=2
2023-01-11T22:40:19.8230064Z dist init r=0, world=2
2023-01-11T22:40:19.8230165Z ok (3.310s)
2023-01-11T22:40:19.8230572Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8231011Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85083
2023-01-11T22:40:19.8231232Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85084
2023-01-11T22:40:19.8231601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8231774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8232428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8232603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8232969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8233149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8233519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8233708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8233951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8234192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8234590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8234988Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8235200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8235435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8235548Z dist init r=0, world=2
2023-01-11T22:40:19.8235655Z dist init r=1, world=2
2023-01-11T22:40:19.8235754Z ok (3.811s)
2023-01-11T22:40:19.8236054Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8236485Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85166
2023-01-11T22:40:19.8236702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85167
2023-01-11T22:40:19.8237054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8237228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8237614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8237806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8238166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8238338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8238707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8238891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8239135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8239359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8239843Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8240302Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8240533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8240762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8241781Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.8241898Z   warnings.warn(
2023-01-11T22:40:19.8242898Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.8243006Z   warnings.warn(
2023-01-11T22:40:19.8243115Z dist init r=0, world=2
2023-01-11T22:40:19.8243224Z dist init r=1, world=2
2023-01-11T22:40:19.8243306Z ok (3.812s)
2023-01-11T22:40:19.8243603Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8244028Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85249
2023-01-11T22:40:19.8244249Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85250
2023-01-11T22:40:19.8244618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8244790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8245169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8245357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8245707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8245878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8246250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8246443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8246689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8246932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8247329Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8247724Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8247951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8248161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8248275Z dist init r=0, world=2
2023-01-11T22:40:19.8248382Z dist init r=1, world=2
2023-01-11T22:40:19.8248537Z ok (3.811s)
2023-01-11T22:40:19.8248888Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8249327Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85332
2023-01-11T22:40:19.8249543Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85333
2023-01-11T22:40:19.8249912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8250067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8250442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8250631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8250996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8251170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8251540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8251726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8251969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8252192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8252593Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8252988Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8253224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8253449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8254474Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.8254586Z   warnings.warn(
2023-01-11T22:40:19.8255590Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.8255707Z   warnings.warn(
2023-01-11T22:40:19.8255816Z dist init r=1, world=2
2023-01-11T22:40:19.8255924Z dist init r=0, world=2
2023-01-11T22:40:19.8256005Z ok (3.812s)
2023-01-11T22:40:19.8256236Z   test_state_dict_rank0_offload_save_load_flow_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8256547Z Tests saving a model checkpoint only on rank 0 and loading it only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85415
2023-01-11T22:40:19.8256765Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85416
2023-01-11T22:40:19.8257134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8257307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8257815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8258012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8258381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8258536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8258907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8259094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8259340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8259582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8259987Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8260382Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8260609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8260838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8260931Z dist init r=1, world=2
2023-01-11T22:40:19.8261038Z dist init r=0, world=2
2023-01-11T22:40:19.8261137Z ok (3.510s)
2023-01-11T22:40:19.8261370Z   test_state_dict_rank0_offload_save_load_flow_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8261677Z Tests saving a model checkpoint only on rank 0 and loading it only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85494
2023-01-11T22:40:19.8261900Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85495
2023-01-11T22:40:19.8262273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8262429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8262807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8262998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8263360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8263532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8263903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8264087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8264342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8264586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8264969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8265368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8265599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8265826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8265936Z dist init r=0, world=2
2023-01-11T22:40:19.8266044Z dist init r=1, world=2
2023-01-11T22:40:19.8266142Z ok (3.511s)
2023-01-11T22:40:19.8266585Z   test_state_dict_save_load_flow_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85573
2023-01-11T22:40:19.8266794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85574
2023-01-11T22:40:19.8267165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8267337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8267710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8267898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8268257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8268430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8268806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8268993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8269218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8269462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8269866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8270260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8270487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8270711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8271741Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.8272084Z   warnings.warn(
2023-01-11T22:40:19.8273108Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.8273226Z   warnings.warn(
2023-01-11T22:40:19.8273465Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:40:19.8273686Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:40:19.8274433Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.8275167Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.8275539Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:40:19.8275777Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:40:19.8275889Z dist init r=0, world=2
2023-01-11T22:40:19.8275995Z dist init r=1, world=2
2023-01-11T22:40:19.8276091Z ok (3.812s)
2023-01-11T22:40:19.8276446Z   test_state_dict_save_load_flow_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85656
2023-01-11T22:40:19.8276668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85657
2023-01-11T22:40:19.8277026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8277202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8277581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8277778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8278140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8278310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8278678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8278864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8279093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8279335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8279733Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8280134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8280362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8280589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8281613Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.8281725Z   warnings.warn(
2023-01-11T22:40:19.8282723Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.8282836Z   warnings.warn(
2023-01-11T22:40:19.8283072Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:40:19.8283310Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:40:19.8284046Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.8284888Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.8285134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:40:19.8285368Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:40:19.8285480Z dist init r=1, world=2
2023-01-11T22:40:19.8285588Z dist init r=0, world=2
2023-01-11T22:40:19.8285686Z ok (3.811s)
2023-01-11T22:40:19.8286028Z   test_state_dict_save_load_flow_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85739
2023-01-11T22:40:19.8286244Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85740
2023-01-11T22:40:19.8286610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8286784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8287161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8287354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8287716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8287886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8288256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8288442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8288690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8288919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8289319Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8289717Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8289943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8290170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8291187Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.8291306Z   warnings.warn(
2023-01-11T22:40:19.8292303Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.8292461Z   warnings.warn(
2023-01-11T22:40:19.8292701Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:40:19.8292938Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:40:19.8293789Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.8294534Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:40:19.8294755Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:40:19.8294985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:40:19.8295095Z dist init r=0, world=2
2023-01-11T22:40:19.8295203Z dist init r=1, world=2
2023-01-11T22:40:19.8295306Z ok (3.810s)
2023-01-11T22:40:19.8295683Z   test_state_dict_skip_module_state_dict_type_local_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85822
2023-01-11T22:40:19.8295902Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85823
2023-01-11T22:40:19.8296274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8296430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8296804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8296990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8297353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8297532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8297904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8298088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8298334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8298580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8298961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8299358Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8299588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8299821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8299931Z dist init r=1, world=2
2023-01-11T22:40:19.8300037Z dist init r=0, world=2
2023-01-11T22:40:19.8300136Z ok (3.812s)
2023-01-11T22:40:19.8300509Z   test_state_dict_skip_module_state_dict_type_sharded_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85905
2023-01-11T22:40:19.8300713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85906
2023-01-11T22:40:19.8301082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8301255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8301630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8301873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8302279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8302456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8302831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8303000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8303244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8303484Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8303881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8304284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8304516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8304746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8305372Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.8305484Z   warnings.warn(
2023-01-11T22:40:19.8306105Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.8306197Z   warnings.warn(
2023-01-11T22:40:19.8306309Z dist init r=0, world=2
2023-01-11T22:40:19.8306421Z dist init r=1, world=2
2023-01-11T22:40:19.8306522Z ok (3.812s)
2023-01-11T22:40:19.8306887Z   test_state_dict_skip_module_state_dict_type_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85988
2023-01-11T22:40:19.8307106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85989
2023-01-11T22:40:19.8307473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8307646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8308004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8308194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8308556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8308735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8309104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8309288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8309532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8309774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8310169Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8310549Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8310777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8311116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8311231Z dist init r=1, world=2
2023-01-11T22:40:19.8311340Z dist init r=0, world=2
2023-01-11T22:40:19.8311439Z ok (3.813s)
2023-01-11T22:40:19.8311737Z   test_state_dict_type (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86071
2023-01-11T22:40:19.8312189Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86072
2023-01-11T22:40:19.8312564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8312738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8313116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8313304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8313671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8313842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8314217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8314404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8314632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8314872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8315269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8315663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8315897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8316125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8316235Z dist init r=0, world=2
2023-01-11T22:40:19.8316343Z dist init r=1, world=2
2023-01-11T22:40:19.8316425Z ok (3.309s)
2023-01-11T22:40:19.8316827Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86150
2023-01-11T22:40:19.8317046Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86151
2023-01-11T22:40:19.8317413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8317586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8317969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8318157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8318515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8318687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8319041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8319227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8319468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8319709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8320251Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8320656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8320886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8321113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8322138Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.8322254Z   warnings.warn(
2023-01-11T22:40:19.8323262Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.8323370Z   warnings.warn(
2023-01-11T22:40:19.8323977Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.8324086Z   warnings.warn(
2023-01-11T22:40:19.8324705Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.8324817Z   warnings.warn(
2023-01-11T22:40:19.8324929Z dist init r=0, world=2
2023-01-11T22:40:19.8325037Z dist init r=1, world=2
2023-01-11T22:40:19.8325136Z ok (3.311s)
2023-01-11T22:40:19.8325540Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86229
2023-01-11T22:40:19.8325742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86230
2023-01-11T22:40:19.8326108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8326282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8326657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8326852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8327213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8327384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8327754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8327941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8328170Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8328414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8328811Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8329303Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8329581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8329810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8330534Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2023-01-11T22:40:19.8330647Z   warnings.warn(
2023-01-11T22:40:19.8331366Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2023-01-11T22:40:19.8331478Z   warnings.warn(
2023-01-11T22:40:19.8331572Z dist init r=0, world=2
2023-01-11T22:40:19.8331678Z dist init r=1, world=2
2023-01-11T22:40:19.8331777Z ok (3.310s)
2023-01-11T22:40:19.8332168Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86308
2023-01-11T22:40:19.8332382Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86309
2023-01-11T22:40:19.8332745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8332913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8333283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8333463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8333813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8333981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8334349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8334531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8334775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8335019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8335420Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8335805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8336036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8336262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8337279Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.8337392Z   warnings.warn(
2023-01-11T22:40:19.8338432Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.8338583Z   warnings.warn(
2023-01-11T22:40:19.8339206Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.8339315Z   warnings.warn(
2023-01-11T22:40:19.8339927Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:40:19.8340037Z   warnings.warn(
2023-01-11T22:40:19.8340149Z dist init r=1, world=2
2023-01-11T22:40:19.8340239Z dist init r=0, world=2
2023-01-11T22:40:19.8340338Z ok (3.211s)
2023-01-11T22:40:19.8340733Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86387
2023-01-11T22:40:19.8340951Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86388
2023-01-11T22:40:19.8341319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8341493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8341868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8342055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8342407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8342579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8342947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8343130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8343373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8343615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8344014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8344411Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8344643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8344854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8345572Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2023-01-11T22:40:19.8345684Z   warnings.warn(
2023-01-11T22:40:19.8346392Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2023-01-11T22:40:19.8346499Z   warnings.warn(
2023-01-11T22:40:19.8346667Z dist init r=0, world=2
2023-01-11T22:40:19.8346813Z dist init r=1, world=2
2023-01-11T22:40:19.8346915Z ok (3.311s)
2023-01-11T22:40:19.8347309Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86466
2023-01-11T22:40:19.8347512Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86467
2023-01-11T22:40:19.8347884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8348056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8348430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8348618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8348981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8349157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8349527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8349711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8349939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8350177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8350571Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8350964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8351197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8351427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8352757Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.8352875Z   warnings.warn(
2023-01-11T22:40:19.8353885Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.8354001Z   warnings.warn(
2023-01-11T22:40:19.8354111Z dist init r=1, world=2
2023-01-11T22:40:19.8354201Z dist init r=0, world=2
2023-01-11T22:40:19.8354300Z ok (3.311s)
2023-01-11T22:40:19.8354693Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86545
2023-01-11T22:40:19.8354913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86546
2023-01-11T22:40:19.8355284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8355456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8355913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8356161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8356511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8356687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8357056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8357241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8357487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8357727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8358127Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8358526Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8358754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8358964Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8359677Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2023-01-11T22:40:19.8359788Z   warnings.warn(
2023-01-11T22:40:19.8360505Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2023-01-11T22:40:19.8360620Z   warnings.warn(
2023-01-11T22:40:19.8360730Z dist init r=0, world=2
2023-01-11T22:40:19.8360838Z dist init r=1, world=2
2023-01-11T22:40:19.8360936Z ok (3.311s)
2023-01-11T22:40:19.8361324Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86624
2023-01-11T22:40:19.8361529Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86625
2023-01-11T22:40:19.8361896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8362070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8362443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8362636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8362996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8363166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8363534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8363721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8363948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8364191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8364590Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8365079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8365311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8365537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8366554Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.8366665Z   warnings.warn(
2023-01-11T22:40:19.8367662Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.8367773Z   warnings.warn(
2023-01-11T22:40:19.8367883Z dist init r=0, world=2
2023-01-11T22:40:19.8367974Z dist init r=1, world=2
2023-01-11T22:40:19.8368073Z ok (3.311s)
2023-01-11T22:40:19.8368462Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86703
2023-01-11T22:40:19.8368681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86704
2023-01-11T22:40:19.8369055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8369230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8369604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8369791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8370133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8370306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8370679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8370863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8371106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8371356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8371755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8372152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8372379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8372589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8373301Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2023-01-11T22:40:19.8373470Z   warnings.warn(
2023-01-11T22:40:19.8374238Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:571: UserWarning: Trying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: Linear(in_features=4, out_features=4, bias=True)
2023-01-11T22:40:19.8374355Z   warnings.warn(
2023-01-11T22:40:19.8374466Z dist init r=1, world=2
2023-01-11T22:40:19.8374571Z dist init r=0, world=2
2023-01-11T22:40:19.8374670Z ok (3.310s)
2023-01-11T22:40:19.8374954Z   test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8375256Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86782
2023-01-11T22:40:19.8375477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86783
2023-01-11T22:40:19.8375851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8376025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8376405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8376592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8376950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8377121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8377487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8377658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8377901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8378149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8378546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8378938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8379166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8379392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8379500Z dist init r=1, world=2
2023-01-11T22:40:19.8379589Z dist init r=0, world=2
2023-01-11T22:40:19.8379686Z ok (3.510s)
2023-01-11T22:40:19.8379970Z   test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8380290Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86861
2023-01-11T22:40:19.8380507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86862
2023-01-11T22:40:19.8380881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8381057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8381433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8381622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8381967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8382138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8382606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8382796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8383040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8383285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8383687Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8384082Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8384293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8384518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8384636Z dist init r=1, world=2
2023-01-11T22:40:19.8384744Z dist init r=0, world=2
2023-01-11T22:40:19.8384840Z ok (3.210s)
2023-01-11T22:40:19.8385112Z   test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8385430Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86940
2023-01-11T22:40:19.8385647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86941
2023-01-11T22:40:19.8386018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8386174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8386552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8386750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8387117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8387289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8387658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8387842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8388087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8388311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8388707Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8389109Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8389337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8389563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8389673Z dist init r=1, world=2
2023-01-11T22:40:19.8389779Z dist init r=0, world=2
2023-01-11T22:40:19.8389876Z ok (3.410s)
2023-01-11T22:40:19.8390131Z   test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T22:40:19.8390446Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87019
2023-01-11T22:40:19.8390664Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87020
2023-01-11T22:40:19.8391078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8391296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8391672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8392086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8392470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8392641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8392993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8393176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8393420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8393673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8394072Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8394463Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8394691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8394918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8395030Z dist init r=0, world=2
2023-01-11T22:40:19.8395120Z dist init r=1, world=2
2023-01-11T22:40:19.8395218Z ok (3.410s)
2023-01-11T22:40:19.8395581Z   test_state_dict_with_shared_parameters_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87098
2023-01-11T22:40:19.8395799Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87099
2023-01-11T22:40:19.8396167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8396341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8396715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8396902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8397248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8397417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8397775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8397963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8398208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8398449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8398846Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8399240Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8399468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8399677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8399787Z dist init r=0, world=2
2023-01-11T22:40:19.8399894Z dist init r=1, world=2
2023-01-11T22:40:19.8400118Z ok (3.409s)
2023-01-11T22:40:19.8400527Z   test_state_dict_with_shared_parameters_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87177
2023-01-11T22:40:19.8400750Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87178
2023-01-11T22:40:19.8401125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8401297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8401657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8401846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8402206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8402380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8402752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8402938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8403184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8403422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8403804Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8404200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8404426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8404652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8404762Z dist init r=1, world=2
2023-01-11T22:40:19.8404869Z dist init r=0, world=2
2023-01-11T22:40:19.8404967Z ok (3.409s)
2023-01-11T22:40:19.8405323Z   test_state_dict_with_shared_parameters_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87256
2023-01-11T22:40:19.8405524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87257
2023-01-11T22:40:19.8405895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8406061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8406437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8406625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8406996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8407168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8407535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8407720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8407949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8408190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8408589Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8408982Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8409306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8409538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8409648Z dist init r=1, world=2
2023-01-11T22:40:19.8409757Z dist init r=0, world=2
2023-01-11T22:40:19.8409838Z ok (3.409s)
2023-01-11T22:40:19.8410142Z   test_wrong_state_dict_config (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87335
2023-01-11T22:40:19.8410356Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87336
2023-01-11T22:40:19.8410726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8410898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8411277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8411468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8411828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:40:19.8412000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:40:19.8412356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:40:19.8412542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:40:19.8412785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:40:19.8413026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:40:19.8413424Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8413830Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:40:19.8414058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:40:19.8414282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:40:19.8415305Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.8415418Z   warnings.warn(
2023-01-11T22:40:19.8416428Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:40:19.8416521Z   warnings.warn(
2023-01-11T22:40:19.8416631Z dist init r=0, world=2
2023-01-11T22:40:19.8416737Z dist init r=1, world=2
2023-01-11T22:40:19.8416836Z ok (3.309s)
2023-01-11T22:40:19.8416858Z 
2023-01-11T22:40:19.8417127Z ----------------------------------------------------------------------
2023-01-11T22:40:19.8417242Z Ran 116 tests in 397.483s
2023-01-11T22:40:19.8417262Z 
2023-01-11T22:40:19.8417353Z OK
2023-01-11T22:40:19.8417372Z 
2023-01-11T22:40:19.8417494Z Generating XML reports...
2023-01-11T22:40:19.8417975Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20230111223341.xml
2023-01-11T22:40:19.8418032Z 
2023-01-11T22:40:19.8418471Z ##[endgroup]
2023-01-11T22:40:19.8418956Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_state_dict (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_state_dict_2vavclr5)
2023-01-11T22:40:19.8418976Z 
2023-01-11T22:40:19.8419223Z Running distributed/test_c10d_nccl ... [2023-01-11 22:40:19.734117]
2023-01-11T22:40:19.8419716Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:40:19.734372]
2023-01-11T22:56:14.6126354Z 
2023-01-11T22:56:14.6128930Z Expand the folded group to see the log file of distributed/test_c10d_nccl
2023-01-11T22:56:14.6129857Z ##[group]PRINTING LOG FILE of distributed/test_c10d_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_nccl_hu8yk7hw)
2023-01-11T22:56:14.6131895Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_all_reduce_coalesced_nccl>, <__main__.CommTest testMethod=test_broadcast_coalesced_nccl>, <__main__.CommTest testMethod=test_nccl_barrier>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids_function_argument>, <__main__.CommTest testMethod=test_nccl_barrier_timeout>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group_non_member>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_detail>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_info>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_off>, <__main__.CommTest testMethod=test_nncl_rank_membership>, <__main__.CommTest testMethod=test_pass_nccl_options_high_priority_stream>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_nccl>, <__main__.CommTest testMethod=test_sequence_num_set_nccl_new_group>, <__main__.CommTest testMethod=test_tensor_dtype_complex>, <__main__.CommTest testMethod=test_tensor_dtype_mismatch>]>
2023-01-11T22:56:14.6133835Z test_all_reduce_coalesced_nccl (__main__.CommTest)
2023-01-11T22:56:14.6134173Z test_broadcast_coalesced_nccl (__main__.CommTest)
2023-01-11T22:56:14.6134494Z test_nccl_barrier (__main__.CommTest)
2023-01-11T22:56:14.6134818Z test_nccl_barrier_device_ids (__main__.CommTest)
2023-01-11T22:56:14.6135157Z test_nccl_barrier_device_ids_function_argument (__main__.CommTest)
2023-01-11T22:56:14.6135610Z test_nccl_barrier_timeout (__main__.CommTest)
2023-01-11T22:56:14.6138082Z test_nccl_barrier_timeout_new_group (__main__.CommTest)
2023-01-11T22:56:14.6138477Z test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest)
2023-01-11T22:56:14.6138869Z test_nccl_warn_not_in_group_debug_detail (__main__.CommTest)
2023-01-11T22:56:14.6139261Z test_nccl_warn_not_in_group_debug_info (__main__.CommTest)
2023-01-11T22:56:14.6139594Z test_nccl_warn_not_in_group_debug_off (__main__.CommTest)
2023-01-11T22:56:14.6139926Z test_nncl_rank_membership (__main__.CommTest)
2023-01-11T22:56:14.6140283Z test_pass_nccl_options_high_priority_stream (__main__.CommTest)
2023-01-11T22:56:14.6140642Z test_sequence_num_incremented_nccl_default (__main__.CommTest)
2023-01-11T22:56:14.6141014Z test_sequence_num_incremented_nccl_subgroup (__main__.CommTest)
2023-01-11T22:56:14.6141380Z test_sequence_num_set_default_pg_nccl (__main__.CommTest)
2023-01-11T22:56:14.6142322Z test_sequence_num_set_nccl_new_group (__main__.CommTest)
2023-01-11T22:56:14.6143121Z test_tensor_dtype_complex (__main__.CommTest)
2023-01-11T22:56:14.6143847Z test_tensor_dtype_mismatch (__main__.CommTest)
2023-01-11T22:56:14.6145471Z <unittest.suite.TestSuite tests=[<__main__.CompilerTest testMethod=test_allgather_into_tensor_work_wait_gpu>, <__main__.CompilerTest testMethod=test_allgather_work_wait_gpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_gpu>, <__main__.CompilerTest testMethod=test_alltoall_work_wait_gpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_gpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_gpu>, <__main__.CompilerTest testMethod=test_nested_comm_tensor_wrapping>, <__main__.CompilerTest testMethod=test_reduce_scatter_tensor_work_wait_gpu>, <__main__.CompilerTest testMethod=test_reduce_scatter_work_wait_gpu>, <__main__.CompilerTest testMethod=test_scatter_work_wait_gpu>]>
2023-01-11T22:56:14.6147294Z test_allgather_into_tensor_work_wait_gpu (__main__.CompilerTest)
2023-01-11T22:56:14.6147668Z test_allgather_work_wait_gpu (__main__.CompilerTest)
2023-01-11T22:56:14.6148019Z test_allreduce_work_wait_gpu (__main__.CompilerTest)
2023-01-11T22:56:14.6148343Z test_alltoall_work_wait_gpu (__main__.CompilerTest)
2023-01-11T22:56:14.6148687Z test_broadcast_work_wait_gpu (__main__.CompilerTest)
2023-01-11T22:56:14.6149052Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest)
2023-01-11T22:56:14.6149604Z test_nested_comm_tensor_wrapping (__main__.CompilerTest)
2023-01-11T22:56:14.6150286Z test_reduce_scatter_tensor_work_wait_gpu (__main__.CompilerTest)
2023-01-11T22:56:14.6151056Z test_reduce_scatter_work_wait_gpu (__main__.CompilerTest)
2023-01-11T22:56:14.6152535Z test_scatter_work_wait_gpu (__main__.CompilerTest)
2023-01-11T22:56:14.6167202Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module>, <__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_channels_last_contig>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_with_then_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_multi_device_module_config>, <__main__.DistributedDataParallelTest testMethod=test_ddp_packed_sequence>, <__main__.DistributedDataParallelTest testMethod=test_ddp_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_with_lazy_parameters>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl_is_view>, <__main__.DistributedDataParallelTest testMethod=test_failure_recovery>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_fp16>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_fp16_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_1devicemodule_1replicaperprocess>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_2devicemodule>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_ids_not_allowed>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_empty_device_ids>, <__main__.DistributedDataParallelTest testMethod=test_nccl_propagate_error_reason>, <__main__.DistributedDataParallelTest testMethod=test_no_grad>, <__main__.DistributedDataParallelTest testMethod=test_param_layout_mismatch_error>, <__main__.DistributedDataParallelTest testMethod=test_pass_default_pg>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]>
2023-01-11T22:56:14.6184750Z test_accumulate_gradients_module (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6185648Z test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6186521Z test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6187398Z test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6189465Z test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6190283Z test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6191300Z test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6192690Z test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6193539Z test_channels_last_contig (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6194316Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6195224Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6196067Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6196946Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6197883Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6198926Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6199878Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6200881Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6201769Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6202626Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6203567Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6204523Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6205505Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6206332Z test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6207226Z test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6208166Z test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6209037Z test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6209905Z test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6210733Z test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6211525Z test_ddp_packed_sequence (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6212260Z test_ddp_weight_sharing (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6213031Z test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6213818Z test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6214644Z test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6215412Z test_failure_recovery (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6216246Z test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6217063Z test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6217922Z test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6218863Z test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6219886Z test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6220849Z test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6221632Z test_fp16 (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6222398Z test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6223223Z test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6224210Z test_fp16_grad_is_view (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6225079Z test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6225938Z test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6226713Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6227491Z test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6228340Z test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6229224Z test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6230157Z test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6231027Z test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6231804Z test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6233141Z test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6233967Z test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6234834Z test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6235750Z test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6236598Z test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6237284Z test_no_grad (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6238028Z test_param_layout_mismatch_error (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6238781Z test_pass_default_pg (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6239479Z test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6240443Z test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6241221Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6241665Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6242029Z <unittest.suite.TestSuite tests=[]>
2023-01-11T22:56:14.6243313Z <unittest.suite.TestSuite tests=[<__main__.NcclErrorHandlingTest testMethod=test_invalid_nccl_blocking_wait_env>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_blocking_wait_with_barrier>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_abort>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_clean_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_nonzero_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigkill>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigterm>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_nonblocking>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_timeout>]>
2023-01-11T22:56:14.6244584Z test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest)
2023-01-11T22:56:14.6244997Z test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest)
2023-01-11T22:56:14.6245395Z test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest)
2023-01-11T22:56:14.6245777Z test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest)
2023-01-11T22:56:14.6246181Z test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest)
2023-01-11T22:56:14.6246583Z test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest)
2023-01-11T22:56:14.6246963Z test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest)
2023-01-11T22:56:14.6247361Z test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest)
2023-01-11T22:56:14.6247727Z test_nccl_timeout (__main__.NcclErrorHandlingTest)
2023-01-11T22:56:14.6249116Z <unittest.suite.TestSuite tests=[<__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_all_to_all_single>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_allgather_base>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_allreduce_coalesced>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_collectives>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_reduce_scatter_base>]>
2023-01-11T22:56:14.6250383Z test_all_to_all_single (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2023-01-11T22:56:14.6250896Z test_allgather_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2023-01-11T22:56:14.6251428Z test_allreduce_coalesced (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2023-01-11T22:56:14.6251953Z test_collectives (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2023-01-11T22:56:14.6252459Z test_reduce_scatter_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2023-01-11T22:56:14.6253016Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLNoGPUTest testMethod=test_init_no_gpus>]>
2023-01-11T22:56:14.6253472Z test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest)
2023-01-11T22:56:14.6255530Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allreduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_barrier>, <__main__.ProcessGroupNCCLTest testMethod=test_broadcast_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_empty_tensors>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_nccl_dist_backend_error>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_send_recv>]>
2023-01-11T22:56:14.6257663Z test_allgather_base_basics (__main__.ProcessGroupNCCLTest)
2023-01-11T22:56:14.6258024Z test_allgather_base_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T22:56:14.6258384Z test_allgather_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T22:56:14.6258735Z test_allreduce_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T22:56:14.6259078Z test_barrier (__main__.ProcessGroupNCCLTest)
2023-01-11T22:56:14.6259403Z test_broadcast_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T22:56:14.6259759Z test_empty_tensors (__main__.ProcessGroupNCCLTest)
2023-01-11T22:56:14.6260109Z test_gather_checks (__main__.ProcessGroupNCCLTest)
2023-01-11T22:56:14.6260439Z test_gather_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T22:56:14.6260781Z test_gather_stress (__main__.ProcessGroupNCCLTest)
2023-01-11T22:56:14.6261147Z test_nccl_dist_backend_error (__main__.ProcessGroupNCCLTest)
2023-01-11T22:56:14.6261486Z test_reduce_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T22:56:14.6261849Z test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest)
2023-01-11T22:56:14.6262234Z test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T22:56:14.6262589Z test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T22:56:14.6262947Z test_scatter_checks (__main__.ProcessGroupNCCLTest)
2023-01-11T22:56:14.6263294Z test_scatter_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T22:56:14.6263639Z test_scatter_stress (__main__.ProcessGroupNCCLTest)
2023-01-11T22:56:14.6263963Z test_send_recv (__main__.ProcessGroupNCCLTest)
2023-01-11T22:56:14.6264519Z <unittest.suite.TestSuite tests=[<__main__.RendezvousEnvTest testMethod=test_common_errors>]>
2023-01-11T22:56:14.6264933Z test_common_errors (__main__.RendezvousEnvTest)
2023-01-11T22:56:14.6265237Z <unittest.suite.TestSuite tests=[]>
2023-01-11T22:56:14.6265652Z <unittest.suite.TestSuite tests=[<__main__.TimeoutTest testMethod=test_default_store_timeout_nccl>]>
2023-01-11T22:56:14.6266070Z test_default_store_timeout_nccl (__main__.TimeoutTest)
2023-01-11T22:56:14.6266751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6267202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6267774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6268243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6268475Z 
2023-01-11T22:56:14.6268573Z Running tests...
2023-01-11T22:56:14.6268974Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6269501Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6269965Z   test_all_reduce_coalesced_nccl (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6270427Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87482
2023-01-11T22:56:14.6270874Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87483
2023-01-11T22:56:14.6271476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6272402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6273046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6273508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6274096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6274524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6275089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6275549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6275969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6276457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6276944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6277419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6278065Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6278755Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6279809Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:56:14.6280436Z   warnings.warn(
2023-01-11T22:56:14.6281295Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:56:14.6282035Z   warnings.warn(
2023-01-11T22:56:14.6282268Z ok (5.432s)
2023-01-11T22:56:14.6282471Z 
2023-01-11T22:56:14.6282752Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6283061Z Ran 1 test in 5.432s
2023-01-11T22:56:14.6283218Z 
2023-01-11T22:56:14.6283309Z OK
2023-01-11T22:56:14.6283439Z 
2023-01-11T22:56:14.6283561Z Generating XML reports...
2023-01-11T22:56:14.6284085Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224023.xml
2023-01-11T22:56:14.6284741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6285187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6285754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6286204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6286494Z 
2023-01-11T22:56:14.6286642Z Running tests...
2023-01-11T22:56:14.6287099Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6287817Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6288389Z   test_broadcast_coalesced_nccl (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6288880Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87599
2023-01-11T22:56:14.6289410Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87600
2023-01-11T22:56:14.6290138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6290606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6291295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6291841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6292491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6292990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6293644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6294175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6294637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6295202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6295804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6296389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6297080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6297856Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6298367Z ok (5.432s)
2023-01-11T22:56:14.6298552Z 
2023-01-11T22:56:14.6298857Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6324042Z Ran 1 test in 5.432s
2023-01-11T22:56:14.6324270Z 
2023-01-11T22:56:14.6324375Z OK
2023-01-11T22:56:14.6324516Z 
2023-01-11T22:56:14.6324630Z Generating XML reports...
2023-01-11T22:56:14.6325246Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224031.xml
2023-01-11T22:56:14.6325920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6326613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6327192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6327670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6327901Z 
2023-01-11T22:56:14.6328010Z Running tests...
2023-01-11T22:56:14.6328394Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6328918Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6329379Z   test_nccl_barrier (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6329824Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87716
2023-01-11T22:56:14.6330260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87717
2023-01-11T22:56:14.6330866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6331320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6331895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6332357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6332932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6333375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6333925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6334386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6334826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6335309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6335688Z skip: Need at least 4 CUDA devices (3.914s)
2023-01-11T22:56:14.6335879Z 
2023-01-11T22:56:14.6336147Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6336468Z Ran 1 test in 3.914s
2023-01-11T22:56:14.6336626Z 
2023-01-11T22:56:14.6336716Z OK (skipped=1)
2023-01-11T22:56:14.6336865Z 
2023-01-11T22:56:14.6336986Z Generating XML reports...
2023-01-11T22:56:14.6337529Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224038.xml
2023-01-11T22:56:14.6338191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6338624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6339201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6339676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6339905Z 
2023-01-11T22:56:14.6339994Z Running tests...
2023-01-11T22:56:14.6340389Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6340915Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6341387Z   test_nccl_barrier_device_ids (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6341832Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87819
2023-01-11T22:56:14.6342287Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87820
2023-01-11T22:56:14.6342891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6343317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6344029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6344502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6345075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6345497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6346061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6346518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6346955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6347433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6347919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6348416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6349058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6349749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6350143Z ok (5.364s)
2023-01-11T22:56:14.6350289Z 
2023-01-11T22:56:14.6350554Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6350858Z Ran 1 test in 5.365s
2023-01-11T22:56:14.6351015Z 
2023-01-11T22:56:14.6351106Z OK
2023-01-11T22:56:14.6351237Z 
2023-01-11T22:56:14.6351360Z Generating XML reports...
2023-01-11T22:56:14.6352389Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224045.xml
2023-01-11T22:56:14.6353196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6353646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6354217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6354671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6354902Z 
2023-01-11T22:56:14.6355010Z Running tests...
2023-01-11T22:56:14.6355411Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6355918Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6356420Z   test_nccl_barrier_device_ids_function_argument (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6356902Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87936
2023-01-11T22:56:14.6357361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87937
2023-01-11T22:56:14.6357948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6358393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6358964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6359419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6359994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6360436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6360998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6361439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6362056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6362552Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6363037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6363499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6364162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6364849Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6365223Z ok (3.923s)
2023-01-11T22:56:14.6365373Z 
2023-01-11T22:56:14.6365638Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6365963Z Ran 1 test in 3.924s
2023-01-11T22:56:14.6366128Z 
2023-01-11T22:56:14.6366220Z OK
2023-01-11T22:56:14.6366333Z 
2023-01-11T22:56:14.6366454Z Generating XML reports...
2023-01-11T22:56:14.6366991Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224052.xml
2023-01-11T22:56:14.6367648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6368075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6368649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6369114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6369341Z 
2023-01-11T22:56:14.6369449Z Running tests...
2023-01-11T22:56:14.6369825Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6370356Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6370827Z   test_nccl_barrier_timeout (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6371266Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88043
2023-01-11T22:56:14.6371718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88044
2023-01-11T22:56:14.6372324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6372773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6373324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6373792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6374369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6374818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6375366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6375830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6376268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6376725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6377110Z skip: Need at least 4 CUDA devices (3.959s)
2023-01-11T22:56:14.6377299Z 
2023-01-11T22:56:14.6377567Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6377895Z Ran 1 test in 3.960s
2023-01-11T22:56:14.6378038Z 
2023-01-11T22:56:14.6378143Z OK (skipped=1)
2023-01-11T22:56:14.6378295Z 
2023-01-11T22:56:14.6378416Z Generating XML reports...
2023-01-11T22:56:14.6379078Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224059.xml
2023-01-11T22:56:14.6379733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6380183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6380752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6381224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6381435Z 
2023-01-11T22:56:14.6381544Z Running tests...
2023-01-11T22:56:14.6381942Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6382467Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6382939Z   test_nccl_barrier_timeout_new_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6383416Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88146
2023-01-11T22:56:14.6383869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88147
2023-01-11T22:56:14.6384473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6384906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6385474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6385946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6386506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6386949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6387515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6387981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6388404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6388880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6389271Z skip: Need at least 4 CUDA devices (3.958s)
2023-01-11T22:56:14.6389465Z 
2023-01-11T22:56:14.6389714Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6390039Z Ran 1 test in 3.959s
2023-01-11T22:56:14.6390197Z 
2023-01-11T22:56:14.6390303Z OK (skipped=1)
2023-01-11T22:56:14.6390452Z 
2023-01-11T22:56:14.6390575Z Generating XML reports...
2023-01-11T22:56:14.6391098Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224105.xml
2023-01-11T22:56:14.6391764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6392627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6393195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6393666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6393896Z 
2023-01-11T22:56:14.6394003Z Running tests...
2023-01-11T22:56:14.6394402Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6394912Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6395416Z   test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6395897Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88249
2023-01-11T22:56:14.6396506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88250
2023-01-11T22:56:14.6397109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6397561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6398126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6398577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6399148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6399593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6400155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6400655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6401102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6401573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6401945Z skip: Need at least 4 CUDA devices (3.964s)
2023-01-11T22:56:14.6402136Z 
2023-01-11T22:56:14.6402403Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6402726Z Ran 1 test in 3.964s
2023-01-11T22:56:14.6402883Z 
2023-01-11T22:56:14.6402988Z OK (skipped=1)
2023-01-11T22:56:14.6403123Z 
2023-01-11T22:56:14.6403245Z Generating XML reports...
2023-01-11T22:56:14.6403783Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224111.xml
2023-01-11T22:56:14.6404443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6404872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6405449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6405912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6406140Z 
2023-01-11T22:56:14.6406248Z Running tests...
2023-01-11T22:56:14.6406628Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6407150Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6407640Z   test_nccl_warn_not_in_group_debug_detail (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6408098Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88352
2023-01-11T22:56:14.6408553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88353
2023-01-11T22:56:14.6409153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6409602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6410154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6410622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6411198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6411639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6412190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6412653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6413089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6413687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6414179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6414681Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6415347Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6416025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6416561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:56:14.6417053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:56:14.6417705Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:56:14.6418384Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:56:14.6418777Z ok (5.473s)
2023-01-11T22:56:14.6418922Z 
2023-01-11T22:56:14.6419183Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6419486Z Ran 1 test in 5.473s
2023-01-11T22:56:14.6419645Z 
2023-01-11T22:56:14.6419737Z OK
2023-01-11T22:56:14.6419866Z 
2023-01-11T22:56:14.6419991Z Generating XML reports...
2023-01-11T22:56:14.6420533Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224118.xml
2023-01-11T22:56:14.6421176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6421619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6422199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6422653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6422881Z 
2023-01-11T22:56:14.6422988Z Running tests...
2023-01-11T22:56:14.6423384Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6423904Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6424376Z   test_nccl_warn_not_in_group_debug_info (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6424844Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88484
2023-01-11T22:56:14.6425293Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88485
2023-01-11T22:56:14.6425880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6426327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6426904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6427367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6427923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6428372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6428940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6429382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6429819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6430307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6430898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6431370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6432331Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6432881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:56:14.6433544Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6434060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:56:14.6434707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:56:14.6435390Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:56:14.6435789Z ok (5.457s)
2023-01-11T22:56:14.6435920Z 
2023-01-11T22:56:14.6436188Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6436512Z Ran 1 test in 5.458s
2023-01-11T22:56:14.6436670Z 
2023-01-11T22:56:14.6436760Z OK
2023-01-11T22:56:14.6436872Z 
2023-01-11T22:56:14.6436995Z Generating XML reports...
2023-01-11T22:56:14.6437531Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224125.xml
2023-01-11T22:56:14.6438194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6438641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6439198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6439670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6439898Z 
2023-01-11T22:56:14.6440005Z Running tests...
2023-01-11T22:56:14.6440388Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6440913Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6441399Z   test_nccl_warn_not_in_group_debug_off (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6441869Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88607
2023-01-11T22:56:14.6442304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88608
2023-01-11T22:56:14.6442906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6443353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6443911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6444381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6444959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6445402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6446009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6446473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6446907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6447388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6447875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6448520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6449196Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6449717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:56:14.6450361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6450888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:56:14.6451537Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:56:14.6452207Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:56:14.6452606Z ok (5.361s)
2023-01-11T22:56:14.6452752Z 
2023-01-11T22:56:14.6453017Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6453321Z Ran 1 test in 5.362s
2023-01-11T22:56:14.6453483Z 
2023-01-11T22:56:14.6453574Z OK
2023-01-11T22:56:14.6453705Z 
2023-01-11T22:56:14.6453832Z Generating XML reports...
2023-01-11T22:56:14.6454370Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224133.xml
2023-01-11T22:56:14.6455015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6455465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6456034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6456486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6456719Z 
2023-01-11T22:56:14.6456825Z Running tests...
2023-01-11T22:56:14.6457225Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6457748Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6458205Z   test_nncl_rank_membership (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6458664Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88730
2023-01-11T22:56:14.6459117Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88731
2023-01-11T22:56:14.6459705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6460150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6460721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6461183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6461747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6462190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6462753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6463214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6463637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6464133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6464618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6465080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6465874Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6466414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:56:14.6467062Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6467573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:56:14.6468219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:56:14.6468899Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:56:14.6469289Z ok (3.966s)
2023-01-11T22:56:14.6469418Z 
2023-01-11T22:56:14.6469683Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6470011Z Ran 1 test in 3.967s
2023-01-11T22:56:14.6470169Z 
2023-01-11T22:56:14.6470266Z OK
2023-01-11T22:56:14.6470397Z 
2023-01-11T22:56:14.6470502Z Generating XML reports...
2023-01-11T22:56:14.6471044Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224141.xml
2023-01-11T22:56:14.6471707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6472554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6473127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6473593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6473821Z 
2023-01-11T22:56:14.6473927Z Running tests...
2023-01-11T22:56:14.6474311Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6474846Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6475347Z   test_pass_nccl_options_high_priority_stream (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6475828Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88839
2023-01-11T22:56:14.6476262Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88840
2023-01-11T22:56:14.6476862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6477310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6477861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6478330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6478906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6479352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6479901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6480364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6480797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6481289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6481757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6482242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6482903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6483580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:56:14.6484248Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6484773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:56:14.6485418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:56:14.6486090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:56:14.6486479Z ok (5.345s)
2023-01-11T22:56:14.6486626Z 
2023-01-11T22:56:14.6486890Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6487211Z Ran 1 test in 5.345s
2023-01-11T22:56:14.6487355Z 
2023-01-11T22:56:14.6487445Z OK
2023-01-11T22:56:14.6487578Z 
2023-01-11T22:56:14.6487702Z Generating XML reports...
2023-01-11T22:56:14.6488241Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224147.xml
2023-01-11T22:56:14.6488890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6489334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6489907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6490376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6490589Z 
2023-01-11T22:56:14.6490695Z Running tests...
2023-01-11T22:56:14.6491096Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6491622Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6492117Z   test_sequence_num_incremented_nccl_default (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6492598Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88960
2023-01-11T22:56:14.6493047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88961
2023-01-11T22:56:14.6493649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6494079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6494646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6495114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6495673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6496123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6496687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6497146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6497568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6498061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6498546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6499011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6499667Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6500416Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6501056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:56:14.6501536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:56:14.6502189Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:56:14.6502873Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:56:14.6503268Z ok (4.964s)
2023-01-11T22:56:14.6503396Z 
2023-01-11T22:56:14.6503663Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6503985Z Ran 1 test in 4.964s
2023-01-11T22:56:14.6504145Z 
2023-01-11T22:56:14.6504235Z OK
2023-01-11T22:56:14.6504367Z 
2023-01-11T22:56:14.6504472Z Generating XML reports...
2023-01-11T22:56:14.6505024Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224155.xml
2023-01-11T22:56:14.6505686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6506138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6506692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6507157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6507383Z 
2023-01-11T22:56:14.6507489Z Running tests...
2023-01-11T22:56:14.6507866Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6508390Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6508897Z   test_sequence_num_incremented_nccl_subgroup (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6509388Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89082
2023-01-11T22:56:14.6509825Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89083
2023-01-11T22:56:14.6510423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6510870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6511445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6512228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6512829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6513272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6513835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6514304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6514742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6515214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6515589Z skip: Need at least 4 CUDA devices (3.970s)
2023-01-11T22:56:14.6515781Z 
2023-01-11T22:56:14.6516047Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6516374Z Ran 1 test in 3.971s
2023-01-11T22:56:14.6516534Z 
2023-01-11T22:56:14.6516623Z OK (skipped=1)
2023-01-11T22:56:14.6516774Z 
2023-01-11T22:56:14.6516899Z Generating XML reports...
2023-01-11T22:56:14.6517441Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224202.xml
2023-01-11T22:56:14.6518256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6518696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6519266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6519732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6519962Z 
2023-01-11T22:56:14.6520051Z Running tests...
2023-01-11T22:56:14.6520449Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6520970Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6521463Z   test_sequence_num_set_default_pg_nccl (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6521921Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89185
2023-01-11T22:56:14.6522379Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89186
2023-01-11T22:56:14.6522987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6523420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6523992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6524458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6525034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6525462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6526027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6526488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6526934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6527408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6527894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6528378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6529021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6529713Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6530106Z ok (4.861s)
2023-01-11T22:56:14.6530252Z 
2023-01-11T22:56:14.6530516Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6530833Z Ran 1 test in 4.861s
2023-01-11T22:56:14.6530992Z 
2023-01-11T22:56:14.6531084Z OK
2023-01-11T22:56:14.6531215Z 
2023-01-11T22:56:14.6531340Z Generating XML reports...
2023-01-11T22:56:14.6531865Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224208.xml
2023-01-11T22:56:14.6532526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6532977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6533550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6534003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6534229Z 
2023-01-11T22:56:14.6534336Z Running tests...
2023-01-11T22:56:14.6534733Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6535349Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6535844Z   test_sequence_num_set_nccl_new_group (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6536317Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89301
2023-01-11T22:56:14.6536768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89302
2023-01-11T22:56:14.6537357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6537807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6538376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6538825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6539398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6539847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6540413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6540861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6541299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6541795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6542278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6542741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6543401Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6543940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:56:14.6544569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6545106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:56:14.6545752Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:56:14.6546434Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:56:14.6546808Z ok (4.865s)
2023-01-11T22:56:14.6546955Z 
2023-01-11T22:56:14.6547216Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6547537Z Ran 1 test in 4.866s
2023-01-11T22:56:14.6547695Z 
2023-01-11T22:56:14.6547787Z OK
2023-01-11T22:56:14.6547903Z 
2023-01-11T22:56:14.6548028Z Generating XML reports...
2023-01-11T22:56:14.6548568Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224215.xml
2023-01-11T22:56:14.6549224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6549654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6550225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6550696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6550922Z 
2023-01-11T22:56:14.6551029Z Running tests...
2023-01-11T22:56:14.6551409Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6552197Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6552849Z   test_tensor_dtype_complex (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6553298Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89421
2023-01-11T22:56:14.6553749Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89422
2023-01-11T22:56:14.6554372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6554819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6555374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6555839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6556415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6556840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6557418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6557881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6558317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6558792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6559275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6559758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6560417Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6561100Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6561501Z ok (5.429s)
2023-01-11T22:56:14.6561650Z 
2023-01-11T22:56:14.6561917Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6562223Z Ran 1 test in 5.429s
2023-01-11T22:56:14.6562382Z 
2023-01-11T22:56:14.6562474Z OK
2023-01-11T22:56:14.6562604Z 
2023-01-11T22:56:14.6562727Z Generating XML reports...
2023-01-11T22:56:14.6563267Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224223.xml
2023-01-11T22:56:14.6563913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6564360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6564935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6565390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6565621Z 
2023-01-11T22:56:14.6565733Z Running tests...
2023-01-11T22:56:14.6566133Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6566656Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6567110Z   test_tensor_dtype_mismatch (__main__.CommTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6567572Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89538
2023-01-11T22:56:14.6568030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89539
2023-01-11T22:56:14.6568614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6569063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6569632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6570207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6570773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6571219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6571788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6572231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6572667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6573157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6573639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6574105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6574770Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6575458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6576509Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:56:14.6577138Z   warnings.warn(
2023-01-11T22:56:14.6578019Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:56:14.6578657Z   warnings.warn(
2023-01-11T22:56:14.6579526Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:56:14.6580127Z   warnings.warn(
2023-01-11T22:56:14.6580996Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:56:14.6581610Z   warnings.warn(
2023-01-11T22:56:14.6581844Z ok (4.880s)
2023-01-11T22:56:14.6581990Z 
2023-01-11T22:56:14.6582239Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6582560Z Ran 1 test in 4.880s
2023-01-11T22:56:14.6582723Z 
2023-01-11T22:56:14.6582813Z OK
2023-01-11T22:56:14.6582948Z 
2023-01-11T22:56:14.6583072Z Generating XML reports...
2023-01-11T22:56:14.6583597Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224230.xml
2023-01-11T22:56:14.6584258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6584705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6585262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6585732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6585962Z 
2023-01-11T22:56:14.6586069Z Running tests...
2023-01-11T22:56:14.6586466Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6586971Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6587602Z   test_allgather_into_tensor_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6588095Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89647
2023-01-11T22:56:14.6588530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89648
2023-01-11T22:56:14.6589135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6589581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6590149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6590597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6591173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6591623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6592440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6592915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6593352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6593824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6594295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6594796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6595461Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6596164Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6597078Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6597804Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6598662Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6599376Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6599682Z ok (5.356s)
2023-01-11T22:56:14.6599829Z 
2023-01-11T22:56:14.6600099Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6600427Z Ran 1 test in 5.357s
2023-01-11T22:56:14.6600638Z 
2023-01-11T22:56:14.6600729Z OK
2023-01-11T22:56:14.6600843Z 
2023-01-11T22:56:14.6600964Z Generating XML reports...
2023-01-11T22:56:14.6601521Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224237.xml
2023-01-11T22:56:14.6602194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6602627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6603197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6603662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6603891Z 
2023-01-11T22:56:14.6603997Z Running tests...
2023-01-11T22:56:14.6604378Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6605068Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6605564Z   test_allgather_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6606018Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89762
2023-01-11T22:56:14.6606469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89763
2023-01-11T22:56:14.6607074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6607522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6608074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6608543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6609122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6609544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6610112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6610571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6611007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6611465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6611949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6612448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6613110Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6613791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6614713Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6615431Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6616280Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6616978Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6617308Z ok (5.474s)
2023-01-11T22:56:14.6617457Z 
2023-01-11T22:56:14.6617730Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6618031Z Ran 1 test in 5.474s
2023-01-11T22:56:14.6618192Z 
2023-01-11T22:56:14.6618282Z OK
2023-01-11T22:56:14.6618413Z 
2023-01-11T22:56:14.6618535Z Generating XML reports...
2023-01-11T22:56:14.6619089Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224245.xml
2023-01-11T22:56:14.6619751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6620199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6620771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6621240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6621527Z 
2023-01-11T22:56:14.6621637Z Running tests...
2023-01-11T22:56:14.6622084Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6622614Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6623082Z   test_allreduce_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6623555Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89877
2023-01-11T22:56:14.6624010Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89878
2023-01-11T22:56:14.6624616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6625047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6625619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6626090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6626647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6627086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6627650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6628108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6628529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6628997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6629486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6629971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6630646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6631342Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6632530Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6633260Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6634098Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6634828Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6635156Z ok (5.356s)
2023-01-11T22:56:14.6635303Z 
2023-01-11T22:56:14.6635548Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6635869Z Ran 1 test in 5.357s
2023-01-11T22:56:14.6636028Z 
2023-01-11T22:56:14.6636121Z OK
2023-01-11T22:56:14.6636254Z 
2023-01-11T22:56:14.6636376Z Generating XML reports...
2023-01-11T22:56:14.6636917Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224253.xml
2023-01-11T22:56:14.6637588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6638036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6638596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6639225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6639458Z 
2023-01-11T22:56:14.6639566Z Running tests...
2023-01-11T22:56:14.6639966Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6640471Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6640955Z   test_alltoall_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6641419Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89992
2023-01-11T22:56:14.6641851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89993
2023-01-11T22:56:14.6642456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6642902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6643475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6643930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6644502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6644945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6645514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6645969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6646405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6646877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6647351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6647861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6648519Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6649211Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6650122Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6650848Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6651703Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6652423Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6652732Z ok (5.456s)
2023-01-11T22:56:14.6652879Z 
2023-01-11T22:56:14.6653143Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6653463Z Ran 1 test in 5.457s
2023-01-11T22:56:14.6653622Z 
2023-01-11T22:56:14.6653712Z OK
2023-01-11T22:56:14.6653826Z 
2023-01-11T22:56:14.6653955Z Generating XML reports...
2023-01-11T22:56:14.6654507Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224301.xml
2023-01-11T22:56:14.6655179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6655612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6656248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6656764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6656994Z 
2023-01-11T22:56:14.6657100Z Running tests...
2023-01-11T22:56:14.6657482Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6658005Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6658493Z   test_broadcast_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6658947Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90109
2023-01-11T22:56:14.6659403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90110
2023-01-11T22:56:14.6660007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6660462Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6661021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6661482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6662052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6662493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6663042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6663502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6663939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6664389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6664880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6665383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6666054Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6666733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6667661Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6668373Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6669227Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6669928Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6670255Z ok (5.342s)
2023-01-11T22:56:14.6670404Z 
2023-01-11T22:56:14.6670668Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6670990Z Ran 1 test in 5.342s
2023-01-11T22:56:14.6671131Z 
2023-01-11T22:56:14.6671222Z OK
2023-01-11T22:56:14.6671354Z 
2023-01-11T22:56:14.6671475Z Generating XML reports...
2023-01-11T22:56:14.6672227Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224308.xml
2023-01-11T22:56:14.6672901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6673539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6674124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6674591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6674804Z 
2023-01-11T22:56:14.6674913Z Running tests...
2023-01-11T22:56:14.6675314Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6675835Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6676314Z   test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6676793Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90224
2023-01-11T22:56:14.6677243Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90225
2023-01-11T22:56:14.6677853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6678291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6678861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6679324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6679880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6680323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6680890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6681350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6681770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6682252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6682742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6683246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6683892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6684586Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6685515Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6686239Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6687085Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6687800Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6688651Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6689362Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6690247Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6691014Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6691864Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6692568Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6693414Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6694104Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6694953Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6695659Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6696497Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6697189Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6697507Z ok (5.455s)
2023-01-11T22:56:14.6697656Z 
2023-01-11T22:56:14.6697924Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6698244Z Ran 1 test in 5.455s
2023-01-11T22:56:14.6698389Z 
2023-01-11T22:56:14.6698480Z OK
2023-01-11T22:56:14.6698615Z 
2023-01-11T22:56:14.6698738Z Generating XML reports...
2023-01-11T22:56:14.6699295Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224316.xml
2023-01-11T22:56:14.6699951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6700405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6701031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6701503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6701732Z 
2023-01-11T22:56:14.6701822Z Running tests...
2023-01-11T22:56:14.6702223Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6702745Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6703230Z   test_nested_comm_tensor_wrapping (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6703707Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90339
2023-01-11T22:56:14.6704156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90340
2023-01-11T22:56:14.6704763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6705197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6705775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6706244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6706821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6707367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6707944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6708408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6708830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6709297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6709785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6710283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6710929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6711631Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6712907Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6713634Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6714472Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6715187Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6716039Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6716761Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6717596Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6718308Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6718636Z ok (5.476s)
2023-01-11T22:56:14.6718782Z 
2023-01-11T22:56:14.6719050Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6719354Z Ran 1 test in 5.476s
2023-01-11T22:56:14.6719513Z 
2023-01-11T22:56:14.6719604Z OK
2023-01-11T22:56:14.6719737Z 
2023-01-11T22:56:14.6719859Z Generating XML reports...
2023-01-11T22:56:14.6720423Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224324.xml
2023-01-11T22:56:14.6721081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6721534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6722103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6722555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6722784Z 
2023-01-11T22:56:14.6722889Z Running tests...
2023-01-11T22:56:14.6723283Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6723806Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6724292Z   test_reduce_scatter_tensor_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6724941Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90454
2023-01-11T22:56:14.6725403Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90455
2023-01-11T22:56:14.6725998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6726449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6727024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6727491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6728053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6728498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6729072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6729537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6729959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6730437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6730932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6731416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6732076Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6732768Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6733700Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6734405Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6735258Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6735966Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6736816Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6737532Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6738365Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6739076Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6739398Z ok (5.343s)
2023-01-11T22:56:14.6739544Z 
2023-01-11T22:56:14.6739806Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6740111Z Ran 1 test in 5.343s
2023-01-11T22:56:14.6740270Z 
2023-01-11T22:56:14.6740360Z OK
2023-01-11T22:56:14.6740491Z 
2023-01-11T22:56:14.6740612Z Generating XML reports...
2023-01-11T22:56:14.6741152Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224331.xml
2023-01-11T22:56:14.6741633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6741815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6742196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6742386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6742407Z 
2023-01-11T22:56:14.6742513Z Running tests...
2023-01-11T22:56:14.6742775Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6743086Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6743320Z   test_reduce_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6743542Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90569
2023-01-11T22:56:14.6743767Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90570
2023-01-11T22:56:14.6744140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6744313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6744691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6744879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6745241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6745396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6745767Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6746023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6746259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6746483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6746729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6746973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6747379Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6747780Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6748399Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6748692Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6749320Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6749603Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6750231Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6750512Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6751243Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6751533Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6751634Z ok (5.439s)
2023-01-11T22:56:14.6751654Z 
2023-01-11T22:56:14.6752160Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6752281Z Ran 1 test in 5.439s
2023-01-11T22:56:14.6752302Z 
2023-01-11T22:56:14.6752376Z OK
2023-01-11T22:56:14.6752395Z 
2023-01-11T22:56:14.6752520Z Generating XML reports...
2023-01-11T22:56:14.6752939Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224339.xml
2023-01-11T22:56:14.6753312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6753489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6753866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6754056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6754076Z 
2023-01-11T22:56:14.6754183Z Running tests...
2023-01-11T22:56:14.6754426Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6754740Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6754975Z   test_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6755194Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90684
2023-01-11T22:56:14.6755409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90685
2023-01-11T22:56:14.6755785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6755958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6756334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6756522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6756867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6757038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6757411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6757597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6757834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6758062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6758311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6758556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6758942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6759341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6759973Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6760416Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6761059Z /opt/conda/lib/python3.10/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T22:56:14.6761344Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T22:56:14.6761445Z ok (5.371s)
2023-01-11T22:56:14.6761465Z 
2023-01-11T22:56:14.6761726Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6761838Z Ran 1 test in 5.372s
2023-01-11T22:56:14.6761859Z 
2023-01-11T22:56:14.6761949Z OK
2023-01-11T22:56:14.6761968Z 
2023-01-11T22:56:14.6762074Z Generating XML reports...
2023-01-11T22:56:14.6762477Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224347.xml
2023-01-11T22:56:14.6762853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6763028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6763405Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6763595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6763614Z 
2023-01-11T22:56:14.6763720Z Running tests...
2023-01-11T22:56:14.6763979Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6764492Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6764773Z   test_accumulate_gradients_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6764992Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90801
2023-01-11T22:56:14.6765215Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90802
2023-01-11T22:56:14.6765590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6765764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6766140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6766329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6766689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6766842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6767213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6767406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6767638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6767881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6768109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6768348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6768752Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6769151Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6769395Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxumty0w6
2023-01-11T22:56:14.6769776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxumty0w6/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6770039Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt201h3oe
2023-01-11T22:56:14.6770313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt201h3oe/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6770547Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6770777Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6770881Z ok (5.947s)
2023-01-11T22:56:14.6770901Z 
2023-01-11T22:56:14.6771166Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6771259Z Ran 1 test in 5.948s
2023-01-11T22:56:14.6771297Z 
2023-01-11T22:56:14.6771369Z OK
2023-01-11T22:56:14.6771388Z 
2023-01-11T22:56:14.6771511Z Generating XML reports...
2023-01-11T22:56:14.6771979Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224354.xml
2023-01-11T22:56:14.6772349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6772523Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6772899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6773088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6773108Z 
2023-01-11T22:56:14.6773213Z Running tests...
2023-01-11T22:56:14.6773458Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6773769Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6774080Z   test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6774305Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90922
2023-01-11T22:56:14.6774522Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90923
2023-01-11T22:56:14.6774891Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6775065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6775441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6775630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6775976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6776148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6776532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6776718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6776946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6777191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6777415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6777655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6778042Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6778442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6778808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3r_6b0og
2023-01-11T22:56:14.6779081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3r_6b0og/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6779341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj4aherfy
2023-01-11T22:56:14.6779611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj4aherfy/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6779847Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6780082Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6780181Z ok (5.825s)
2023-01-11T22:56:14.6780201Z 
2023-01-11T22:56:14.6780453Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6780563Z Ran 1 test in 5.825s
2023-01-11T22:56:14.6780584Z 
2023-01-11T22:56:14.6780679Z OK
2023-01-11T22:56:14.6780699Z 
2023-01-11T22:56:14.6780824Z Generating XML reports...
2023-01-11T22:56:14.6781288Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224403.xml
2023-01-11T22:56:14.6781657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6781831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6782209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6782381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6782420Z 
2023-01-11T22:56:14.6782509Z Running tests...
2023-01-11T22:56:14.6782767Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6783080Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6783374Z   test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6783595Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91043
2023-01-11T22:56:14.6783812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91044
2023-01-11T22:56:14.6784178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6784350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6784709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6784897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6785261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6785440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6785816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6786003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6786235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6786485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6786699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6786940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6787341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6787792Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6788114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp187_4dcv
2023-01-11T22:56:14.6788435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp187_4dcv/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6788693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp6vb5fup
2023-01-11T22:56:14.6788967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp6vb5fup/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6789205Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6789417Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6789519Z ok (5.872s)
2023-01-11T22:56:14.6789539Z 
2023-01-11T22:56:14.6789803Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6789919Z Ran 1 test in 5.872s
2023-01-11T22:56:14.6789939Z 
2023-01-11T22:56:14.6790034Z OK
2023-01-11T22:56:14.6790053Z 
2023-01-11T22:56:14.6790176Z Generating XML reports...
2023-01-11T22:56:14.6790639Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224411.xml
2023-01-11T22:56:14.6791008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6791182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6791543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6791733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6791753Z 
2023-01-11T22:56:14.6792098Z Running tests...
2023-01-11T22:56:14.6792383Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6792712Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6793017Z   test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6793237Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91164
2023-01-11T22:56:14.6793452Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91165
2023-01-11T22:56:14.6793806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6793983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6794359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6794549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6794913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6795088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6795465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6795650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6795881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6796112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6796338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6796576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6796980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6797537Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6797807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp90twa3w5
2023-01-11T22:56:14.6798079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp90twa3w5/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6798340Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwcfuk622
2023-01-11T22:56:14.6798608Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwcfuk622/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6798826Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6799066Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6799166Z ok (5.862s)
2023-01-11T22:56:14.6799189Z 
2023-01-11T22:56:14.6799463Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6799574Z Ran 1 test in 5.862s
2023-01-11T22:56:14.6799594Z 
2023-01-11T22:56:14.6799687Z OK
2023-01-11T22:56:14.6799706Z 
2023-01-11T22:56:14.6799830Z Generating XML reports...
2023-01-11T22:56:14.6800294Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224419.xml
2023-01-11T22:56:14.6800706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6800882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6801264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6801455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6801474Z 
2023-01-11T22:56:14.6801646Z Running tests...
2023-01-11T22:56:14.6801913Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6802222Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6802509Z   test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6802727Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91285
2023-01-11T22:56:14.6802927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91286
2023-01-11T22:56:14.6803293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6803467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6803842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6804030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6804399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6804571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6804942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6805110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6805340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6805586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6805816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6806053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6806560Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6807125Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.6807526Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6808076Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.6808341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9i7q2n7v
2023-01-11T22:56:14.6808615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9i7q2n7v/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6808853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfvsnved0
2023-01-11T22:56:14.6809127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfvsnved0/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6809232Z ok (5.463s)
2023-01-11T22:56:14.6809252Z 
2023-01-11T22:56:14.6809515Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6809624Z Ran 1 test in 5.463s
2023-01-11T22:56:14.6809645Z 
2023-01-11T22:56:14.6809738Z OK
2023-01-11T22:56:14.6809757Z 
2023-01-11T22:56:14.6809879Z Generating XML reports...
2023-01-11T22:56:14.6810349Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224427.xml
2023-01-11T22:56:14.6810725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6810883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6811261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6811451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6811471Z 
2023-01-11T22:56:14.6811577Z Running tests...
2023-01-11T22:56:14.6811839Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6812150Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6812435Z   test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6812653Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91406
2023-01-11T22:56:14.6812861Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91407
2023-01-11T22:56:14.6813234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6813410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6813788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6813977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6814339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6814512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6814883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6815170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6815391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6815637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6815868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6816106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6816510Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6817065Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.6817470Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6818020Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.6818278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu13057vr
2023-01-11T22:56:14.6818549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu13057vr/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6818807Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuyhjy5nl
2023-01-11T22:56:14.6819073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuyhjy5nl/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6819173Z ok (5.399s)
2023-01-11T22:56:14.6819194Z 
2023-01-11T22:56:14.6819457Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6819566Z Ran 1 test in 5.399s
2023-01-11T22:56:14.6819586Z 
2023-01-11T22:56:14.6819679Z OK
2023-01-11T22:56:14.6819698Z 
2023-01-11T22:56:14.6819820Z Generating XML reports...
2023-01-11T22:56:14.6820282Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224435.xml
2023-01-11T22:56:14.6820652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6820809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6821183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6821381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6821400Z 
2023-01-11T22:56:14.6821506Z Running tests...
2023-01-11T22:56:14.6821772Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6822085Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6822365Z   test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6822581Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91527
2023-01-11T22:56:14.6822781Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91528
2023-01-11T22:56:14.6823154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6823327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6823772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6824012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6824375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6824548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6824919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6825104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6825316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6825562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6825791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6826036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6826435Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6826832Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6827093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp48gl845k
2023-01-11T22:56:14.6827367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp48gl845k/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6827624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprizyz9ux
2023-01-11T22:56:14.6827879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprizyz9ux/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6827979Z ok (5.333s)
2023-01-11T22:56:14.6828002Z 
2023-01-11T22:56:14.6828270Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6828381Z Ran 1 test in 5.334s
2023-01-11T22:56:14.6828401Z 
2023-01-11T22:56:14.6828492Z OK
2023-01-11T22:56:14.6828512Z 
2023-01-11T22:56:14.6828633Z Generating XML reports...
2023-01-11T22:56:14.6829097Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224443.xml
2023-01-11T22:56:14.6829472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6829629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6830009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6830201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6830220Z 
2023-01-11T22:56:14.6830331Z Running tests...
2023-01-11T22:56:14.6830595Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6830907Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6831203Z   test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6831423Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91648
2023-01-11T22:56:14.6831637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91649
2023-01-11T22:56:14.6832239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6832424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6832812Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6833147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6833520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6833690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6834059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6834245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6834460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6834708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6834936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6835174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6835582Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6835979Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6836240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxvmepb2l
2023-01-11T22:56:14.6836516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxvmepb2l/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6836772Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt0y5s1tn
2023-01-11T22:56:14.6837025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt0y5s1tn/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6837125Z ok (5.349s)
2023-01-11T22:56:14.6837146Z 
2023-01-11T22:56:14.6837408Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6837522Z Ran 1 test in 5.349s
2023-01-11T22:56:14.6837544Z 
2023-01-11T22:56:14.6837635Z OK
2023-01-11T22:56:14.6837654Z 
2023-01-11T22:56:14.6837775Z Generating XML reports...
2023-01-11T22:56:14.6838237Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224450.xml
2023-01-11T22:56:14.6838606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6838779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6839139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6839331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6839351Z 
2023-01-11T22:56:14.6839456Z Running tests...
2023-01-11T22:56:14.6839715Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6840034Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6840308Z   test_channels_last_contig (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6840527Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91769
2023-01-11T22:56:14.6840742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91770
2023-01-11T22:56:14.6841093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6841266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6841639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6841829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6842245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6842481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6842859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6843046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6843276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6843503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6843730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6843969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6844369Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6844771Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6844873Z ok (5.454s)
2023-01-11T22:56:14.6844893Z 
2023-01-11T22:56:14.6845154Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6845263Z Ran 1 test in 5.454s
2023-01-11T22:56:14.6845283Z 
2023-01-11T22:56:14.6845356Z OK
2023-01-11T22:56:14.6845391Z 
2023-01-11T22:56:14.6845501Z Generating XML reports...
2023-01-11T22:56:14.6845963Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224458.xml
2023-01-11T22:56:14.6846327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6846500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6846878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6847072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6847092Z 
2023-01-11T22:56:14.6847199Z Running tests...
2023-01-11T22:56:14.6847459Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6847751Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6847978Z   test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6848347Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6848565Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91886
2023-01-11T22:56:14.6848779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91887
2023-01-11T22:56:14.6849148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6849325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6849701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6849872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6850232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6850406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6850782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6850965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6851198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6851600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6851828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6852068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6852456Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6852856Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6853114Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp44k6ef18
2023-01-11T22:56:14.6853384Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp44k6ef18/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6853637Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9hdu7nj3
2023-01-11T22:56:14.6853913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9hdu7nj3/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6854012Z ok (5.335s)
2023-01-11T22:56:14.6854033Z 
2023-01-11T22:56:14.6854296Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6854409Z Ran 1 test in 5.335s
2023-01-11T22:56:14.6854428Z 
2023-01-11T22:56:14.6854502Z OK
2023-01-11T22:56:14.6854521Z 
2023-01-11T22:56:14.6854645Z Generating XML reports...
2023-01-11T22:56:14.6855105Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224506.xml
2023-01-11T22:56:14.6855475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6855647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6856027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6856223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6856243Z 
2023-01-11T22:56:14.6856349Z Running tests...
2023-01-11T22:56:14.6856589Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6856898Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6857139Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6857406Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6857623Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92006
2023-01-11T22:56:14.6857838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92007
2023-01-11T22:56:14.6858203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6858381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6858759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6858930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6859296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6859465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6859836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6860023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6860252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6860598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6860832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6861062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6861466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6861859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6862119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp53k8hzno
2023-01-11T22:56:14.6862391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp53k8hzno/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6862646Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpje12ualh
2023-01-11T22:56:14.6862923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpje12ualh/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6863023Z ok (5.362s)
2023-01-11T22:56:14.6863043Z 
2023-01-11T22:56:14.6863306Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6863399Z Ran 1 test in 5.362s
2023-01-11T22:56:14.6863418Z 
2023-01-11T22:56:14.6863510Z OK
2023-01-11T22:56:14.6863529Z 
2023-01-11T22:56:14.6863652Z Generating XML reports...
2023-01-11T22:56:14.6864112Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224513.xml
2023-01-11T22:56:14.6864481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6864656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6865034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6865229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6865249Z 
2023-01-11T22:56:14.6865354Z Running tests...
2023-01-11T22:56:14.6865595Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6865906Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6866145Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6866399Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6866617Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92126
2023-01-11T22:56:14.6866835Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92127
2023-01-11T22:56:14.6867206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6867386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6867747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6867935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6868297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6868467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6868836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6869022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6869256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6869503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6869814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6870062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6870469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6870863Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6871121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3efdpva5
2023-01-11T22:56:14.6871395Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3efdpva5/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6871649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3k4_nqz7
2023-01-11T22:56:14.6872164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3k4_nqz7/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6872414Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6872630Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6872862Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6873094Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6874028Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:56:14.6874140Z   warnings.warn(
2023-01-11T22:56:14.6875067Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:56:14.6875181Z   warnings.warn(
2023-01-11T22:56:14.6875420Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6875652Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6875882Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6876113Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6876329Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6876562Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6876786Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6877015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6877115Z ok (5.533s)
2023-01-11T22:56:14.6877136Z 
2023-01-11T22:56:14.6877406Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6877519Z Ran 1 test in 5.533s
2023-01-11T22:56:14.6877538Z 
2023-01-11T22:56:14.6877628Z OK
2023-01-11T22:56:14.6877647Z 
2023-01-11T22:56:14.6877752Z Generating XML reports...
2023-01-11T22:56:14.6878218Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224521.xml
2023-01-11T22:56:14.6878588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6878907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6879298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6879487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6879508Z 
2023-01-11T22:56:14.6879613Z Running tests...
2023-01-11T22:56:14.6879871Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6880165Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6880402Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6880653Z DDP works as expected when layer is checkpointed only once. ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6880872Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92246
2023-01-11T22:56:14.6881092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92247
2023-01-11T22:56:14.6881461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6881635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6882011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6882198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6882548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6882722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6883093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6883280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6883517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6883763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6883992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6884236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6884638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6885018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6885278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmvy2yg2v
2023-01-11T22:56:14.6885554Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmvy2yg2v/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6885815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpul6rh_hn
2023-01-11T22:56:14.6886085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpul6rh_hn/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6886323Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6886557Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6886788Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6887004Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6887982Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:56:14.6888141Z   warnings.warn(
2023-01-11T22:56:14.6889063Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:56:14.6889172Z   warnings.warn(
2023-01-11T22:56:14.6889409Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6889637Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6889867Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6890106Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6890337Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6890563Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6890770Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6890997Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6891096Z ok (5.552s)
2023-01-11T22:56:14.6891117Z 
2023-01-11T22:56:14.6891383Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6891496Z Ran 1 test in 5.553s
2023-01-11T22:56:14.6891515Z 
2023-01-11T22:56:14.6891605Z OK
2023-01-11T22:56:14.6891624Z 
2023-01-11T22:56:14.6891746Z Generating XML reports...
2023-01-11T22:56:14.6892217Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224529.xml
2023-01-11T22:56:14.6892572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6892747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6893126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6893314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6893334Z 
2023-01-11T22:56:14.6893439Z Running tests...
2023-01-11T22:56:14.6893694Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6894009Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6894275Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6894620Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6894843Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92366
2023-01-11T22:56:14.6895059Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92367
2023-01-11T22:56:14.6895426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6895600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6895975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6896163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6896528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6896762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6897162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6897354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6897584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6897832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6898064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6898309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6898714Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6899115Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6899381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph7roflnh
2023-01-11T22:56:14.6899638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph7roflnh/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6899893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmply3awygx
2023-01-11T22:56:14.6900162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmply3awygx/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6902266Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6902502Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6902730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6902962Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6903072Z ok (5.590s)
2023-01-11T22:56:14.6903092Z 
2023-01-11T22:56:14.6903370Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6903463Z Ran 1 test in 5.590s
2023-01-11T22:56:14.6903483Z 
2023-01-11T22:56:14.6903574Z OK
2023-01-11T22:56:14.6903593Z 
2023-01-11T22:56:14.6903715Z Generating XML reports...
2023-01-11T22:56:14.6904181Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224537.xml
2023-01-11T22:56:14.6904549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6904722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6905098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6905287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6905310Z 
2023-01-11T22:56:14.6905402Z Running tests...
2023-01-11T22:56:14.6905664Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6905975Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6906234Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6906586Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6906802Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92486
2023-01-11T22:56:14.6907020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92487
2023-01-11T22:56:14.6907388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6907562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6908041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6908239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6908605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6908777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6909146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6909331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6909561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6909806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6910025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6910270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6910673Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6911071Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6911329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf50__ldv
2023-01-11T22:56:14.6911599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf50__ldv/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6912101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpncm7qzgn
2023-01-11T22:56:14.6912393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpncm7qzgn/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6912637Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6912852Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6913086Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6913317Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6913416Z ok (5.466s)
2023-01-11T22:56:14.6913437Z 
2023-01-11T22:56:14.6913711Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6913821Z Ran 1 test in 5.466s
2023-01-11T22:56:14.6913841Z 
2023-01-11T22:56:14.6913932Z OK
2023-01-11T22:56:14.6913951Z 
2023-01-11T22:56:14.6914073Z Generating XML reports...
2023-01-11T22:56:14.6914521Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224545.xml
2023-01-11T22:56:14.6914898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6915076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6915455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6915644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6915664Z 
2023-01-11T22:56:14.6915770Z Running tests...
2023-01-11T22:56:14.6916030Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6916342Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6916584Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6916944Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6917326Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92606
2023-01-11T22:56:14.6917551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92607
2023-01-11T22:56:14.6917927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6918099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6918475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6918664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6919028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6919182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6919560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6919753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6919985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6920228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6920454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6920694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6921095Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6921494Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6921746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsxa_lg8v
2023-01-11T22:56:14.6922019Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsxa_lg8v/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6922275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxd777mf3
2023-01-11T22:56:14.6922545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxd777mf3/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6922781Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6923011Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6923802Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:56:14.6924593Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:56:14.6924833Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6925066Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6925223Z ok (5.490s)
2023-01-11T22:56:14.6925284Z 
2023-01-11T22:56:14.6925563Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6925658Z Ran 1 test in 5.490s
2023-01-11T22:56:14.6925694Z 
2023-01-11T22:56:14.6925769Z OK
2023-01-11T22:56:14.6925788Z 
2023-01-11T22:56:14.6925912Z Generating XML reports...
2023-01-11T22:56:14.6926377Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224552.xml
2023-01-11T22:56:14.6926748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6926921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6927298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6927488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6927512Z 
2023-01-11T22:56:14.6927622Z Running tests...
2023-01-11T22:56:14.6927867Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6928179Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6928418Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6928793Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6929011Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92726
2023-01-11T22:56:14.6929229Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92727
2023-01-11T22:56:14.6929597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6929770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6930137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6930329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6930697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6930867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6931237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6931424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6931654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6931902Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6932129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6932365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6932770Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6933168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6933430Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxn7hjsya
2023-01-11T22:56:14.6933705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxn7hjsya/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6933960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzn1b1mrh
2023-01-11T22:56:14.6934230Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzn1b1mrh/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6934391Z ok (5.489s)
2023-01-11T22:56:14.6934412Z 
2023-01-11T22:56:14.6934717Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6934815Z Ran 1 test in 5.489s
2023-01-11T22:56:14.6934834Z 
2023-01-11T22:56:14.6934925Z OK
2023-01-11T22:56:14.6934944Z 
2023-01-11T22:56:14.6935067Z Generating XML reports...
2023-01-11T22:56:14.6935532Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224600.xml
2023-01-11T22:56:14.6935903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6936075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6936452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6936643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6936666Z 
2023-01-11T22:56:14.6936756Z Running tests...
2023-01-11T22:56:14.6937022Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6937337Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6937576Z   test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6937847Z Checkpointing should work with static graph in the case of checkpointing ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6938065Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92846
2023-01-11T22:56:14.6938281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92847
2023-01-11T22:56:14.6938650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6938824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6939188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6939383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6939748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6939919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6940288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6940476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6940707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6940953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6941159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6941404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6941805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6942210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6942472Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgb6bthe4
2023-01-11T22:56:14.6942745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgb6bthe4/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6943000Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplu7lehm3
2023-01-11T22:56:14.6943271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplu7lehm3/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6943562Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6943830Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6944065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6944298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6944401Z ok (5.497s)
2023-01-11T22:56:14.6944421Z 
2023-01-11T22:56:14.6944688Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6944798Z Ran 1 test in 5.497s
2023-01-11T22:56:14.6944817Z 
2023-01-11T22:56:14.6944908Z OK
2023-01-11T22:56:14.6944927Z 
2023-01-11T22:56:14.6945049Z Generating XML reports...
2023-01-11T22:56:14.6945496Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224608.xml
2023-01-11T22:56:14.6945871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6946050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6946426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6946616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6946635Z 
2023-01-11T22:56:14.6946742Z Running tests...
2023-01-11T22:56:14.6947002Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6947313Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6947571Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6947825Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6948046Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92966
2023-01-11T22:56:14.6948269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92967
2023-01-11T22:56:14.6948641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6948815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6949192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6949381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6949741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6949894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6950267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6950462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6950697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6950943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6951166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6951403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6951801Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6952467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6952715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpedcgi4d3
2023-01-11T22:56:14.6953136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpedcgi4d3/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6953399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgcvgttlm
2023-01-11T22:56:14.6953672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgcvgttlm/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6954461Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:56:14.6955252Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:56:14.6956191Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:56:14.6956304Z   warnings.warn(
2023-01-11T22:56:14.6957228Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:56:14.6957340Z   warnings.warn(
2023-01-11T22:56:14.6957581Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6957819Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6958054Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6958287Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6958368Z ok (5.434s)
2023-01-11T22:56:14.6958388Z 
2023-01-11T22:56:14.6958657Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6958771Z Ran 1 test in 5.434s
2023-01-11T22:56:14.6958793Z 
2023-01-11T22:56:14.6958886Z OK
2023-01-11T22:56:14.6958905Z 
2023-01-11T22:56:14.6959027Z Generating XML reports...
2023-01-11T22:56:14.6959492Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224616.xml
2023-01-11T22:56:14.6959861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6960038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6960417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6960592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6960612Z 
2023-01-11T22:56:14.6960719Z Running tests...
2023-01-11T22:56:14.6960981Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6961396Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6961659Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6961933Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6962151Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93086
2023-01-11T22:56:14.6962367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93087
2023-01-11T22:56:14.6962720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6962896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6963274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6963468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6963830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6964002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6964371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6964557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6964787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6965018Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6965244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6965483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6965892Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6966290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6966549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf9xflmuv
2023-01-11T22:56:14.6966824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf9xflmuv/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6967081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpufl4x53q
2023-01-11T22:56:14.6967355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpufl4x53q/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6968265Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:56:14.6968384Z   warnings.warn(
2023-01-11T22:56:14.6969309Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:56:14.6969419Z   warnings.warn(
2023-01-11T22:56:14.6969656Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6969890Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6970225Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6970464Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6970565Z ok (5.446s)
2023-01-11T22:56:14.6970586Z 
2023-01-11T22:56:14.6970854Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6970948Z Ran 1 test in 5.446s
2023-01-11T22:56:14.6970967Z 
2023-01-11T22:56:14.6971057Z OK
2023-01-11T22:56:14.6971077Z 
2023-01-11T22:56:14.6971197Z Generating XML reports...
2023-01-11T22:56:14.6971659Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224624.xml
2023-01-11T22:56:14.6972027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6972201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6972582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6972775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6972795Z 
2023-01-11T22:56:14.6972884Z Running tests...
2023-01-11T22:56:14.6973145Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6973455Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6973713Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6973952Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6974169Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93206
2023-01-11T22:56:14.6974386Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93207
2023-01-11T22:56:14.6974757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6974936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6975296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6975486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6975847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6976016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6976386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6976574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6976802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6977056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6977262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6977502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6977903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6978304Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6978562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_z3oejqa
2023-01-11T22:56:14.6978833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_z3oejqa/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6979089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpznsm2vg0
2023-01-11T22:56:14.6979473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpznsm2vg0/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6979714Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6979932Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6980167Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6980399Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6980500Z ok (5.457s)
2023-01-11T22:56:14.6980520Z 
2023-01-11T22:56:14.6980785Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6980895Z Ran 1 test in 5.458s
2023-01-11T22:56:14.6980914Z 
2023-01-11T22:56:14.6981004Z OK
2023-01-11T22:56:14.6981023Z 
2023-01-11T22:56:14.6981145Z Generating XML reports...
2023-01-11T22:56:14.6981600Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224631.xml
2023-01-11T22:56:14.6981972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6982146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6982523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6982712Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6982732Z 
2023-01-11T22:56:14.6982839Z Running tests...
2023-01-11T22:56:14.6983095Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6983407Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6983664Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.6983893Z Test that checkpointing with weight sharing works. ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6984113Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93326
2023-01-11T22:56:14.6984331Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93327
2023-01-11T22:56:14.6984704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6984878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6985256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6985443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6985807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6985967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6986350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6986542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6986773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6987023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6987248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6987486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6987887Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6988337Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6988676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1mxfh16h
2023-01-11T22:56:14.6988951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1mxfh16h/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6989210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjcwqmhlj
2023-01-11T22:56:14.6989483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjcwqmhlj/_remote_module_non_scriptable.py
2023-01-11T22:56:14.6989719Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6989954Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6990187Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6990419Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6990657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6990869Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6991099Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6991329Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.6991431Z ok (5.482s)
2023-01-11T22:56:14.6991452Z 
2023-01-11T22:56:14.6991718Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6992051Z Ran 1 test in 5.482s
2023-01-11T22:56:14.6992081Z 
2023-01-11T22:56:14.6992183Z OK
2023-01-11T22:56:14.6992203Z 
2023-01-11T22:56:14.6992330Z Generating XML reports...
2023-01-11T22:56:14.6992790Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224639.xml
2023-01-11T22:56:14.6993169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6993346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6993723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6993914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6993934Z 
2023-01-11T22:56:14.6994041Z Running tests...
2023-01-11T22:56:14.6994304Z ----------------------------------------------------------------------
2023-01-11T22:56:14.6994615Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.6994909Z   test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.6995112Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93446
2023-01-11T22:56:14.6995337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93447
2023-01-11T22:56:14.6995705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6995879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6996256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6996445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6996808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.6996977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.6997332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.6997664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.6997901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.6998124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.6998371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.6998614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.6999019Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6999420Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.6999680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5xrh98t9
2023-01-11T22:56:14.6999941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5xrh98t9/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7000199Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_tws_pu7
2023-01-11T22:56:14.7000470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_tws_pu7/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7000574Z ok (5.480s)
2023-01-11T22:56:14.7000595Z 
2023-01-11T22:56:14.7000905Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7001017Z Ran 1 test in 5.480s
2023-01-11T22:56:14.7001036Z 
2023-01-11T22:56:14.7001128Z OK
2023-01-11T22:56:14.7001146Z 
2023-01-11T22:56:14.7001270Z Generating XML reports...
2023-01-11T22:56:14.7001716Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224647.xml
2023-01-11T22:56:14.7002084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7002268Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7002647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7002837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7002856Z 
2023-01-11T22:56:14.7002962Z Running tests...
2023-01-11T22:56:14.7003221Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7003533Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7003838Z   test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7004039Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93567
2023-01-11T22:56:14.7004256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93568
2023-01-11T22:56:14.7004635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7004809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7005185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7005373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7005737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7005906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7006277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7006447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7006778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7007029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7007258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7007499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7007906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7008304Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7008563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn9zwk1w1
2023-01-11T22:56:14.7008819Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn9zwk1w1/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7009084Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7xb1fp3h
2023-01-11T22:56:14.7009356Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7xb1fp3h/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7009457Z ok (5.449s)
2023-01-11T22:56:14.7009477Z 
2023-01-11T22:56:14.7009741Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7009853Z Ran 1 test in 5.449s
2023-01-11T22:56:14.7009872Z 
2023-01-11T22:56:14.7009965Z OK
2023-01-11T22:56:14.7009984Z 
2023-01-11T22:56:14.7010105Z Generating XML reports...
2023-01-11T22:56:14.7010572Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224655.xml
2023-01-11T22:56:14.7010926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7011102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7011490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7011680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7011700Z 
2023-01-11T22:56:14.7011805Z Running tests...
2023-01-11T22:56:14.7012069Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7012382Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7012692Z   test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7012912Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93688
2023-01-11T22:56:14.7013113Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93689
2023-01-11T22:56:14.7013482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7013661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7014041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7014229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7014588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7014759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7015127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7015296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7015528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7015871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7016104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7016342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7016743Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7017142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7017401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0co_kw9q
2023-01-11T22:56:14.7017676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0co_kw9q/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7017915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4gjgncj8
2023-01-11T22:56:14.7018193Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4gjgncj8/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7018294Z ok (5.435s)
2023-01-11T22:56:14.7018314Z 
2023-01-11T22:56:14.7018578Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7018688Z Ran 1 test in 5.435s
2023-01-11T22:56:14.7018708Z 
2023-01-11T22:56:14.7018797Z OK
2023-01-11T22:56:14.7018816Z 
2023-01-11T22:56:14.7018938Z Generating XML reports...
2023-01-11T22:56:14.7019405Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224703.xml
2023-01-11T22:56:14.7019757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7019932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7020311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7020507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7020526Z 
2023-01-11T22:56:14.7020627Z Running tests...
2023-01-11T22:56:14.7020896Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7021210Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7021451Z   test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.7021739Z This unit test verifies whether a DDP communication hook that calls allreduce and then ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7021959Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93809
2023-01-11T22:56:14.7022177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93810
2023-01-11T22:56:14.7022545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7022710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7023090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7023279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7023642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7023812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7024184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7024373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7024605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7024884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7025165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7025406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7025812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7026209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7026474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpytpb4m7p
2023-01-11T22:56:14.7026750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpytpb4m7p/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7027007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4mfbi0u5
2023-01-11T22:56:14.7027285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4mfbi0u5/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7027370Z ok (5.471s)
2023-01-11T22:56:14.7027390Z 
2023-01-11T22:56:14.7027654Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7027764Z Ran 1 test in 5.472s
2023-01-11T22:56:14.7027784Z 
2023-01-11T22:56:14.7027875Z OK
2023-01-11T22:56:14.7027895Z 
2023-01-11T22:56:14.7028017Z Generating XML reports...
2023-01-11T22:56:14.7028475Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224710.xml
2023-01-11T22:56:14.7028846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7029021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7029401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7029582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7029602Z 
2023-01-11T22:56:14.7029711Z Running tests...
2023-01-11T22:56:14.7029970Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7030282Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7030510Z   test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.7030802Z This unit test verifies whether the Future object is passed properly using nccl backend. ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7031023Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93930
2023-01-11T22:56:14.7031239Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93931
2023-01-11T22:56:14.7031591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7031772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7032418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7032611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7032976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7033148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7033530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7033719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7033953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7034271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7034555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7034795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7035202Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7035597Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7035858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9bkt82ag
2023-01-11T22:56:14.7036132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9bkt82ag/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7036390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvysx3ju9
2023-01-11T22:56:14.7036652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvysx3ju9/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7036758Z ok (5.349s)
2023-01-11T22:56:14.7036778Z 
2023-01-11T22:56:14.7037045Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7037157Z Ran 1 test in 5.349s
2023-01-11T22:56:14.7037177Z 
2023-01-11T22:56:14.7037270Z OK
2023-01-11T22:56:14.7037289Z 
2023-01-11T22:56:14.7037414Z Generating XML reports...
2023-01-11T22:56:14.7037874Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224718.xml
2023-01-11T22:56:14.7038244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7038419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7038779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7038977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7038996Z 
2023-01-11T22:56:14.7039106Z Running tests...
2023-01-11T22:56:14.7039368Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7039679Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7039968Z   test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7040187Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94051
2023-01-11T22:56:14.7040406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94052
2023-01-11T22:56:14.7040757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7040933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7041319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7041510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7041875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7042047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7042416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7042603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7042836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7043048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7043199Z skip: Need at least 4 CUDA devices (3.862s)
2023-01-11T22:56:14.7043270Z 
2023-01-11T22:56:14.7043595Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7043713Z Ran 1 test in 3.862s
2023-01-11T22:56:14.7043733Z 
2023-01-11T22:56:14.7043840Z OK (skipped=1)
2023-01-11T22:56:14.7043859Z 
2023-01-11T22:56:14.7043985Z Generating XML reports...
2023-01-11T22:56:14.7044446Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224726.xml
2023-01-11T22:56:14.7044816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7044974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7045352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7045542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7045561Z 
2023-01-11T22:56:14.7045673Z Running tests...
2023-01-11T22:56:14.7045939Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7046321Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7046525Z   test_ddp_packed_sequence (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.7046784Z Tests that DDP with ``device_ids`` specified can run a forward and ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7047004Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94154
2023-01-11T22:56:14.7047205Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94155
2023-01-11T22:56:14.7047575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7047749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7048131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7048325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7048690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7048862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7049235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7049406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7049639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7049887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7050115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7050360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7050765Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7051165Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7051424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq60m080r
2023-01-11T22:56:14.7051697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq60m080r/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7051935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphs_hkh73
2023-01-11T22:56:14.7052207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphs_hkh73/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7052308Z ok (6.086s)
2023-01-11T22:56:14.7052328Z 
2023-01-11T22:56:14.7052701Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7052817Z Ran 1 test in 6.086s
2023-01-11T22:56:14.7052841Z 
2023-01-11T22:56:14.7052937Z OK
2023-01-11T22:56:14.7052956Z 
2023-01-11T22:56:14.7053079Z Generating XML reports...
2023-01-11T22:56:14.7053541Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224732.xml
2023-01-11T22:56:14.7053911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7054070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7054449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7054640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7054659Z 
2023-01-11T22:56:14.7054766Z Running tests...
2023-01-11T22:56:14.7055034Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7055348Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7055624Z   test_ddp_weight_sharing (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7055843Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94275
2023-01-11T22:56:14.7056041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94276
2023-01-11T22:56:14.7056411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7056586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7056962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7057153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7057523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7057695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7058069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7058256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7058469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7058718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7058942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7059181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7059590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7059989Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7060250Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6pri3y6s
2023-01-11T22:56:14.7060509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1j559rkf
2023-01-11T22:56:14.7060785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6pri3y6s/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7061036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1j559rkf/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7061277Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7061515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7061846Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7062090Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7062321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7062548Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7062782Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7062995Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7063096Z ok (5.836s)
2023-01-11T22:56:14.7063116Z 
2023-01-11T22:56:14.7063384Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7063496Z Ran 1 test in 5.837s
2023-01-11T22:56:14.7063516Z 
2023-01-11T22:56:14.7063607Z OK
2023-01-11T22:56:14.7063627Z 
2023-01-11T22:56:14.7063755Z Generating XML reports...
2023-01-11T22:56:14.7064224Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224740.xml
2023-01-11T22:56:14.7064597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7064754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7065137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7065329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7065349Z 
2023-01-11T22:56:14.7065457Z Running tests...
2023-01-11T22:56:14.7065718Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7066029Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7066315Z   test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7066539Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94396
2023-01-11T22:56:14.7066756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94397
2023-01-11T22:56:14.7067108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7067285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7067662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7067851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7068218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7068390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7068769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7068957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7069171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7069422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7069651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7069894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7070296Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7070886Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment.
2023-01-11T22:56:14.7071217Z   warnings.warn('Lazy modules are a new feature under heavy development '
2023-01-11T22:56:14.7071616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7072386Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/lazy.py:180: UserWarning: Lazy modules are a new feature under heavy development so changes to the API or functionality can happen at any moment.
2023-01-11T22:56:14.7072677Z   warnings.warn('Lazy modules are a new feature under heavy development '
2023-01-11T22:56:14.7072923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf2fzxq43
2023-01-11T22:56:14.7073198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf2fzxq43/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7073463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp071e1rv
2023-01-11T22:56:14.7073739Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp071e1rv/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7073841Z ok (3.928s)
2023-01-11T22:56:14.7073862Z 
2023-01-11T22:56:14.7074128Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7074242Z Ran 1 test in 3.928s
2023-01-11T22:56:14.7074262Z 
2023-01-11T22:56:14.7074353Z OK
2023-01-11T22:56:14.7074372Z 
2023-01-11T22:56:14.7074476Z Generating XML reports...
2023-01-11T22:56:14.7074941Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224748.xml
2023-01-11T22:56:14.7075310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7075484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7075870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7076067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7076087Z 
2023-01-11T22:56:14.7076196Z Running tests...
2023-01-11T22:56:14.7076457Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7076769Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7077035Z   test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7077257Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94503
2023-01-11T22:56:14.7077474Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94504
2023-01-11T22:56:14.7077842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7078020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7078404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7078595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7078958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7079130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7079487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7079675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7079907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7080156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7080530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7080780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7081190Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7081590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7081834Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu2v90ayl
2023-01-11T22:56:14.7082113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu2v90ayl/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7082373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxtngmyhv
2023-01-11T22:56:14.7082647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxtngmyhv/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7082754Z ok (5.328s)
2023-01-11T22:56:14.7082775Z 
2023-01-11T22:56:14.7083042Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7083154Z Ran 1 test in 5.328s
2023-01-11T22:56:14.7083173Z 
2023-01-11T22:56:14.7083265Z OK
2023-01-11T22:56:14.7083285Z 
2023-01-11T22:56:14.7083408Z Generating XML reports...
2023-01-11T22:56:14.7083852Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224755.xml
2023-01-11T22:56:14.7084222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7084399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7084780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7084971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7084997Z 
2023-01-11T22:56:14.7085108Z Running tests...
2023-01-11T22:56:14.7085370Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7085683Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7085959Z   test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7086181Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94624
2023-01-11T22:56:14.7086397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94625
2023-01-11T22:56:14.7086772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7086947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7087334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7087529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7087889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7088060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7088413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7088600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7088831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7089078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7089309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7089647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7090065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7090462Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7090722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7535o4xt
2023-01-11T22:56:14.7090974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7535o4xt/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7091229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6hkuh8k4
2023-01-11T22:56:14.7091501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6hkuh8k4/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7091603Z ok (5.332s)
2023-01-11T22:56:14.7091627Z 
2023-01-11T22:56:14.7091898Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7092010Z Ran 1 test in 5.333s
2023-01-11T22:56:14.7092029Z 
2023-01-11T22:56:14.7092121Z OK
2023-01-11T22:56:14.7092141Z 
2023-01-11T22:56:14.7092263Z Generating XML reports...
2023-01-11T22:56:14.7092710Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224802.xml
2023-01-11T22:56:14.7093079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7093253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7093629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7093825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7093845Z 
2023-01-11T22:56:14.7093952Z Running tests...
2023-01-11T22:56:14.7094222Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7094533Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7094805Z   test_failure_recovery (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7095009Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94745
2023-01-11T22:56:14.7095226Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94746
2023-01-11T22:56:14.7095594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7095767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7096145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7096335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7096707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7096880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7097235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7097423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7097654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7097876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7098120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7098367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7098871Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7099277Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7099540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30cadq_z
2023-01-11T22:56:14.7099797Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30cadq_z/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7100057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz4nkv2kc
2023-01-11T22:56:14.7100330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz4nkv2kc/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7100570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7100846Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7101100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7101343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7101744Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7102140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7102360Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7102595Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7102696Z ok (5.973s)
2023-01-11T22:56:14.7102717Z 
2023-01-11T22:56:14.7102981Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7103092Z Ran 1 test in 5.973s
2023-01-11T22:56:14.7103114Z 
2023-01-11T22:56:14.7103206Z OK
2023-01-11T22:56:14.7103228Z 
2023-01-11T22:56:14.7103353Z Generating XML reports...
2023-01-11T22:56:14.7103818Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224810.xml
2023-01-11T22:56:14.7104171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7104350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7104729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7104920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7104940Z 
2023-01-11T22:56:14.7105047Z Running tests...
2023-01-11T22:56:14.7105309Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7105621Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7105937Z   test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7106709Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82632 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.628s)
2023-01-11T22:56:14.7106729Z 
2023-01-11T22:56:14.7106993Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7107085Z Ran 1 test in 1.628s
2023-01-11T22:56:14.7107104Z 
2023-01-11T22:56:14.7107210Z OK (skipped=1)
2023-01-11T22:56:14.7107229Z 
2023-01-11T22:56:14.7107353Z Generating XML reports...
2023-01-11T22:56:14.7107815Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224818.xml
2023-01-11T22:56:14.7108299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7108481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7108863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7109055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7109074Z 
2023-01-11T22:56:14.7109181Z Running tests...
2023-01-11T22:56:14.7109424Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7109736Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7110037Z   test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7110792Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/83301 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.610s)
2023-01-11T22:56:14.7110815Z 
2023-01-11T22:56:14.7111075Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7111188Z Ran 1 test in 1.610s
2023-01-11T22:56:14.7111207Z 
2023-01-11T22:56:14.7111315Z OK (skipped=1)
2023-01-11T22:56:14.7111334Z 
2023-01-11T22:56:14.7111457Z Generating XML reports...
2023-01-11T22:56:14.7112153Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224822.xml
2023-01-11T22:56:14.7112531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7112712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7113101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7113291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7113311Z 
2023-01-11T22:56:14.7113419Z Running tests...
2023-01-11T22:56:14.7113679Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7113990Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7114289Z   test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7115038Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82385 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.660s)
2023-01-11T22:56:14.7115061Z 
2023-01-11T22:56:14.7115327Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7115421Z Ran 1 test in 1.660s
2023-01-11T22:56:14.7115440Z 
2023-01-11T22:56:14.7115550Z OK (skipped=1)
2023-01-11T22:56:14.7115569Z 
2023-01-11T22:56:14.7115692Z Generating XML reports...
2023-01-11T22:56:14.7116154Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224826.xml
2023-01-11T22:56:14.7116525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7116702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7117080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7117273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7117383Z 
2023-01-11T22:56:14.7117500Z Running tests...
2023-01-11T22:56:14.7117796Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7118124Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7118447Z   test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7119197Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82979 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.644s)
2023-01-11T22:56:14.7119218Z 
2023-01-11T22:56:14.7119477Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7119589Z Ran 1 test in 1.644s
2023-01-11T22:56:14.7119609Z 
2023-01-11T22:56:14.7119715Z OK (skipped=1)
2023-01-11T22:56:14.7119737Z 
2023-01-11T22:56:14.7119862Z Generating XML reports...
2023-01-11T22:56:14.7120325Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224830.xml
2023-01-11T22:56:14.7120698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7120857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7121235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7121424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7121444Z 
2023-01-11T22:56:14.7121551Z Running tests...
2023-01-11T22:56:14.7121814Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7122125Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7122450Z   test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7123200Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82400 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.655s)
2023-01-11T22:56:14.7123220Z 
2023-01-11T22:56:14.7123477Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7123569Z Ran 1 test in 1.656s
2023-01-11T22:56:14.7123607Z 
2023-01-11T22:56:14.7123695Z OK (skipped=1)
2023-01-11T22:56:14.7123714Z 
2023-01-11T22:56:14.7123837Z Generating XML reports...
2023-01-11T22:56:14.7124295Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224834.xml
2023-01-11T22:56:14.7124672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7124850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7125226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7125417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7125437Z 
2023-01-11T22:56:14.7125545Z Running tests...
2023-01-11T22:56:14.7125787Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7126099Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7126415Z   test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7127215Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82500 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.645s)
2023-01-11T22:56:14.7127274Z 
2023-01-11T22:56:14.7127547Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7127659Z Ran 1 test in 1.645s
2023-01-11T22:56:14.7127678Z 
2023-01-11T22:56:14.7127785Z OK (skipped=1)
2023-01-11T22:56:14.7127804Z 
2023-01-11T22:56:14.7127926Z Generating XML reports...
2023-01-11T22:56:14.7128386Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224838.xml
2023-01-11T22:56:14.7128755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7128911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7129294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7129491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7129511Z 
2023-01-11T22:56:14.7129619Z Running tests...
2023-01-11T22:56:14.7129881Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7130192Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7130439Z   test_fp16 (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7130661Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95081
2023-01-11T22:56:14.7130860Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95082
2023-01-11T22:56:14.7131227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7131402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7131788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7131977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7132344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7132516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7132886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7133073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7133286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7133535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7133772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7134017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7134423Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7134820Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7135081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxs_gz32y
2023-01-11T22:56:14.7135355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxs_gz32y/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7135613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptf00_i5p
2023-01-11T22:56:14.7135867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptf00_i5p/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7136106Z ok (5.868s)
2023-01-11T22:56:14.7136127Z 
2023-01-11T22:56:14.7136399Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7136512Z Ran 1 test in 5.868s
2023-01-11T22:56:14.7136531Z 
2023-01-11T22:56:14.7136623Z OK
2023-01-11T22:56:14.7136642Z 
2023-01-11T22:56:14.7136768Z Generating XML reports...
2023-01-11T22:56:14.7137229Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224842.xml
2023-01-11T22:56:14.7137600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7137757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7138138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7138332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7138358Z 
2023-01-11T22:56:14.7138466Z Running tests...
2023-01-11T22:56:14.7138729Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7139040Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7139329Z   test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7139548Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95202
2023-01-11T22:56:14.7139749Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95203
2023-01-11T22:56:14.7140118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7140294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7140677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7140875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7141244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7141418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7141791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7141977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7142193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7142442Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7142677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7142925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7143330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7143886Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7144279Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7144865Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7145178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpggspb30e
2023-01-11T22:56:14.7145456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpggspb30e/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7145711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpysukri2g
2023-01-11T22:56:14.7145967Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpysukri2g/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7146069Z ok (5.435s)
2023-01-11T22:56:14.7146089Z 
2023-01-11T22:56:14.7146353Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7146469Z Ran 1 test in 5.436s
2023-01-11T22:56:14.7146488Z 
2023-01-11T22:56:14.7146580Z OK
2023-01-11T22:56:14.7146600Z 
2023-01-11T22:56:14.7146723Z Generating XML reports...
2023-01-11T22:56:14.7147194Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224850.xml
2023-01-11T22:56:14.7147565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7147723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7148103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7148295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7148315Z 
2023-01-11T22:56:14.7148425Z Running tests...
2023-01-11T22:56:14.7148689Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7149001Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7149284Z   test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7149512Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95323
2023-01-11T22:56:14.7149731Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95324
2023-01-11T22:56:14.7150083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7150259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7150641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7150831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7151191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7151368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7151748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7152177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7152401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7152650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7152880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7153118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7153534Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7154168Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7154634Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7155188Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7155450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwxn3jfkq
2023-01-11T22:56:14.7155726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwxn3jfkq/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7155988Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6mwib349
2023-01-11T22:56:14.7156240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6mwib349/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7156345Z ok (5.464s)
2023-01-11T22:56:14.7156366Z 
2023-01-11T22:56:14.7156630Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7156742Z Ran 1 test in 5.464s
2023-01-11T22:56:14.7156762Z 
2023-01-11T22:56:14.7156854Z OK
2023-01-11T22:56:14.7156873Z 
2023-01-11T22:56:14.7156998Z Generating XML reports...
2023-01-11T22:56:14.7157462Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224858.xml
2023-01-11T22:56:14.7157835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7158011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7158379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7158572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7158591Z 
2023-01-11T22:56:14.7158697Z Running tests...
2023-01-11T22:56:14.7158959Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7159270Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7159537Z   test_fp16_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7159758Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95444
2023-01-11T22:56:14.7159974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95445
2023-01-11T22:56:14.7160325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7160507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7160887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7161075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7161442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7161615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7161989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7162176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7162408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7162637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7162969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7163217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7163623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7164019Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7164283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiotcblv3
2023-01-11T22:56:14.7164559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiotcblv3/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7164817Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpytxpo2dp
2023-01-11T22:56:14.7165096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpytxpo2dp/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7165183Z ok (5.841s)
2023-01-11T22:56:14.7165202Z 
2023-01-11T22:56:14.7165469Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7165579Z Ran 1 test in 5.841s
2023-01-11T22:56:14.7165598Z 
2023-01-11T22:56:14.7165691Z OK
2023-01-11T22:56:14.7165711Z 
2023-01-11T22:56:14.7165836Z Generating XML reports...
2023-01-11T22:56:14.7166295Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224905.xml
2023-01-11T22:56:14.7166667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7166841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7167202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7167402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7167424Z 
2023-01-11T22:56:14.7167535Z Running tests...
2023-01-11T22:56:14.7167797Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7168109Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7168430Z   test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7168654Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95565
2023-01-11T22:56:14.7168873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95566
2023-01-11T22:56:14.7169243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7169401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7169785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7169981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7170345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7170518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7170893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7171087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7171317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7171546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7171778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7172126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7172538Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7172936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7173195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_u6hd1zk
2023-01-11T22:56:14.7173468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_u6hd1zk/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7173726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb6gk0yq7
2023-01-11T22:56:14.7173997Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb6gk0yq7/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7174219Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7174460Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7174696Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7174933Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7175166Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7175395Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7175622Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7175854Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7176088Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7176311Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7176539Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7176764Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7176993Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7177342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7177612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7177877Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7178138Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7178349Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7178598Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7179029Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7179294Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7179568Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7179830Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7180098Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7180361Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7180667Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7180876Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7181244Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7181563Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7181829Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7182091Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7182352Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7182614Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7182874Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7183086Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7183347Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7183502Z ok (6.859s)
2023-01-11T22:56:14.7183522Z 
2023-01-11T22:56:14.7183869Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7184018Z Ran 1 test in 6.860s
2023-01-11T22:56:14.7184038Z 
2023-01-11T22:56:14.7184142Z OK
2023-01-11T22:56:14.7204439Z 
2023-01-11T22:56:14.7204624Z Generating XML reports...
2023-01-11T22:56:14.7205151Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224913.xml
2023-01-11T22:56:14.7205521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7205684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7206052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7206230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7206261Z 
2023-01-11T22:56:14.7206359Z Running tests...
2023-01-11T22:56:14.7206614Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7206909Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7207177Z   test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7207383Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95686
2023-01-11T22:56:14.7207595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95687
2023-01-11T22:56:14.7207964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7208139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7208520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7208713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7209058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7209232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7209608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7209801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7210027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7210255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7210404Z skip: Need at least 4 CUDA devices (3.953s)
2023-01-11T22:56:14.7210425Z 
2023-01-11T22:56:14.7210936Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7211055Z Ran 1 test in 3.953s
2023-01-11T22:56:14.7211075Z 
2023-01-11T22:56:14.7211165Z OK (skipped=1)
2023-01-11T22:56:14.7211184Z 
2023-01-11T22:56:14.7211307Z Generating XML reports...
2023-01-11T22:56:14.7211772Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224923.xml
2023-01-11T22:56:14.7212138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7212312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7212687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7212875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7212895Z 
2023-01-11T22:56:14.7213001Z Running tests...
2023-01-11T22:56:14.7213269Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7213560Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7213833Z   test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7214047Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95789
2023-01-11T22:56:14.7214259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95790
2023-01-11T22:56:14.7214623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7214796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7215169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7215354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7215707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7215880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7216246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7216427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7216653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7217196Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7217747Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7218277Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7218812Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7219451Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7219994Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7220220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7220751Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7221284Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7221802Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7222338Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7222865Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7223391Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7223498Z ok (3.862s)
2023-01-11T22:56:14.7223519Z 
2023-01-11T22:56:14.7223792Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7223903Z Ran 1 test in 3.862s
2023-01-11T22:56:14.7223923Z 
2023-01-11T22:56:14.7223996Z OK
2023-01-11T22:56:14.7224014Z 
2023-01-11T22:56:14.7224136Z Generating XML reports...
2023-01-11T22:56:14.7224599Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224929.xml
2023-01-11T22:56:14.7224969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7225142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7225515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7225704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7225854Z 
2023-01-11T22:56:14.7225969Z Running tests...
2023-01-11T22:56:14.7226216Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7226526Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7226815Z   test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7227033Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95892
2023-01-11T22:56:14.7227248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95893
2023-01-11T22:56:14.7227611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7227781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7228160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7228351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7228693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7228861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7229228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7229413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7229640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7229884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7230108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7230351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7230754Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7231130Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7231385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3l49crlv
2023-01-11T22:56:14.7231652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3l49crlv/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7232146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6nujmngv
2023-01-11T22:56:14.7232429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6nujmngv/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7232531Z ok (5.852s)
2023-01-11T22:56:14.7232557Z 
2023-01-11T22:56:14.7232837Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7232949Z Ran 1 test in 5.852s
2023-01-11T22:56:14.7232968Z 
2023-01-11T22:56:14.7233041Z OK
2023-01-11T22:56:14.7233079Z 
2023-01-11T22:56:14.7233184Z Generating XML reports...
2023-01-11T22:56:14.7233644Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224935.xml
2023-01-11T22:56:14.7234010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7234183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7234556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7234745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7234764Z 
2023-01-11T22:56:14.7234871Z Running tests...
2023-01-11T22:56:14.7235306Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7235611Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7235917Z   test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7236133Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96013
2023-01-11T22:56:14.7236344Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96014
2023-01-11T22:56:14.7236709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7236880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7237255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7237446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7237795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7237967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7238334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7238519Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7238746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7238988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7239216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7239448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7239853Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7240228Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7240482Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4eq52vtj
2023-01-11T22:56:14.7240747Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4eq52vtj/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7240998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpim1w_ryp
2023-01-11T22:56:14.7241262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpim1w_ryp/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7241362Z ok (5.951s)
2023-01-11T22:56:14.7241382Z 
2023-01-11T22:56:14.7241646Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7241762Z Ran 1 test in 5.951s
2023-01-11T22:56:14.7241781Z 
2023-01-11T22:56:14.7241875Z OK
2023-01-11T22:56:14.7241894Z 
2023-01-11T22:56:14.7242000Z Generating XML reports...
2023-01-11T22:56:14.7242460Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224943.xml
2023-01-11T22:56:14.7242826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7242998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7243372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7243561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7243581Z 
2023-01-11T22:56:14.7243687Z Running tests...
2023-01-11T22:56:14.7243953Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7244359Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7244676Z   test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7244893Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96134
2023-01-11T22:56:14.7245107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96135
2023-01-11T22:56:14.7245473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7245645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7246019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7246207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7246568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7246724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7247092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7247275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7247501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7247744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7247969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7248202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7248598Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7249003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7249241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpomx0oosc
2023-01-11T22:56:14.7249508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpomx0oosc/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7249758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0t_gz7h8
2023-01-11T22:56:14.7250018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0t_gz7h8/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7250250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7250479Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7250579Z ok (5.879s)
2023-01-11T22:56:14.7250602Z 
2023-01-11T22:56:14.7250871Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7250964Z Ran 1 test in 5.879s
2023-01-11T22:56:14.7250983Z 
2023-01-11T22:56:14.7251075Z OK
2023-01-11T22:56:14.7251093Z 
2023-01-11T22:56:14.7251216Z Generating XML reports...
2023-01-11T22:56:14.7251671Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224951.xml
2023-01-11T22:56:14.7252036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7252209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7252585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7252774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7252793Z 
2023-01-11T22:56:14.7252901Z Running tests...
2023-01-11T22:56:14.7253244Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7253561Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7253876Z   test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7254092Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96255
2023-01-11T22:56:14.7254306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96256
2023-01-11T22:56:14.7254672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7254846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7255223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7255393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7255760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7255930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7256303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7256488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7256715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7256957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7257178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7257411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7257799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7258196Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7258448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp39xrxk6t
2023-01-11T22:56:14.7258713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp39xrxk6t/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7258963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0vb42av9
2023-01-11T22:56:14.7259225Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0vb42av9/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7259458Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7259689Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7259779Z ok (5.876s)
2023-01-11T22:56:14.7259817Z 
2023-01-11T22:56:14.7260063Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7260176Z Ran 1 test in 5.877s
2023-01-11T22:56:14.7260196Z 
2023-01-11T22:56:14.7260286Z OK
2023-01-11T22:56:14.7260306Z 
2023-01-11T22:56:14.7260428Z Generating XML reports...
2023-01-11T22:56:14.7260886Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225000.xml
2023-01-11T22:56:14.7261253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7261430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7261804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7261975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7262067Z 
2023-01-11T22:56:14.7262203Z Running tests...
2023-01-11T22:56:14.7262473Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7262781Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7263053Z   test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7263269Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96376
2023-01-11T22:56:14.7263482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96377
2023-01-11T22:56:14.7263846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7264000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7264377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7264570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7264933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7265102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7265470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7265655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7265881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7266107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7266238Z skip: Need at least 4 CUDA devices (3.960s)
2023-01-11T22:56:14.7266258Z 
2023-01-11T22:56:14.7266524Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7266642Z Ran 1 test in 3.960s
2023-01-11T22:56:14.7266662Z 
2023-01-11T22:56:14.7266769Z OK (skipped=1)
2023-01-11T22:56:14.7266787Z 
2023-01-11T22:56:14.7266910Z Generating XML reports...
2023-01-11T22:56:14.7267368Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225008.xml
2023-01-11T22:56:14.7267735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7267908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7268284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7268456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7268475Z 
2023-01-11T22:56:14.7268583Z Running tests...
2023-01-11T22:56:14.7268848Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7269157Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7269431Z   test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7269646Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96479
2023-01-11T22:56:14.7269860Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96480
2023-01-11T22:56:14.7270227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7270381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7270754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7270942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7271409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7271585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7272224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7272420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7272650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7272880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7273011Z skip: Need at least 8 CUDA devices (3.864s)
2023-01-11T22:56:14.7273032Z 
2023-01-11T22:56:14.7273308Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7273418Z Ran 1 test in 3.865s
2023-01-11T22:56:14.7273437Z 
2023-01-11T22:56:14.7273549Z OK (skipped=1)
2023-01-11T22:56:14.7273569Z 
2023-01-11T22:56:14.7273696Z Generating XML reports...
2023-01-11T22:56:14.7274151Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225014.xml
2023-01-11T22:56:14.7274518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7274690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7275045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7275235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7275254Z 
2023-01-11T22:56:14.7275361Z Running tests...
2023-01-11T22:56:14.7275621Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7275929Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7276230Z   test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7276448Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96582
2023-01-11T22:56:14.7276662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96583
2023-01-11T22:56:14.7277028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7277182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7277558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7277744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7278103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7278282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7278653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7278837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7279069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7279297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7279529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7279763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7280165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7280720Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7280830Z ok (4.756s)
2023-01-11T22:56:14.7280850Z 
2023-01-11T22:56:14.7281117Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7281227Z Ran 1 test in 4.756s
2023-01-11T22:56:14.7281246Z 
2023-01-11T22:56:14.7281336Z OK
2023-01-11T22:56:14.7281354Z 
2023-01-11T22:56:14.7281459Z Generating XML reports...
2023-01-11T22:56:14.7281922Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225020.xml
2023-01-11T22:56:14.7282286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7282459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7282833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7283028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7283047Z 
2023-01-11T22:56:14.7283154Z Running tests...
2023-01-11T22:56:14.7283416Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7283711Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7284020Z   test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7284239Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96691
2023-01-11T22:56:14.7284455Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96692
2023-01-11T22:56:14.7284826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7284998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7285385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7285574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7285937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7286091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7286465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7286651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7286881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7287106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7287257Z skip: Need at least 4 CUDA devices (3.982s)
2023-01-11T22:56:14.7287280Z 
2023-01-11T22:56:14.7287543Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7287653Z Ran 1 test in 3.983s
2023-01-11T22:56:14.7287673Z 
2023-01-11T22:56:14.7287778Z OK (skipped=1)
2023-01-11T22:56:14.7287798Z 
2023-01-11T22:56:14.7287902Z Generating XML reports...
2023-01-11T22:56:14.7288365Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225027.xml
2023-01-11T22:56:14.7288732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7288908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7289282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7289468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7289544Z 
2023-01-11T22:56:14.7289700Z Running tests...
2023-01-11T22:56:14.7289974Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7290267Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7290580Z   test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7290797Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96794
2023-01-11T22:56:14.7291013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96795
2023-01-11T22:56:14.7291381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7291555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7291933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7292128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7292491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7292646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7293017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7293200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7293428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7293674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7293900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7294151Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7294554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7294956Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7295200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw97x20xb
2023-01-11T22:56:14.7295470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw97x20xb/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7295723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbi8z0n60
2023-01-11T22:56:14.7295989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbi8z0n60/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7296227Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7296460Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7296561Z ok (5.859s)
2023-01-11T22:56:14.7296581Z 
2023-01-11T22:56:14.7296845Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7296938Z Ran 1 test in 5.859s
2023-01-11T22:56:14.7296957Z 
2023-01-11T22:56:14.7297049Z OK
2023-01-11T22:56:14.7297069Z 
2023-01-11T22:56:14.7297192Z Generating XML reports...
2023-01-11T22:56:14.7297652Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225034.xml
2023-01-11T22:56:14.7298019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7298193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7298570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7298872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7298893Z 
2023-01-11T22:56:14.7299006Z Running tests...
2023-01-11T22:56:14.7299253Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7299571Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7299884Z   test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7300101Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96915
2023-01-11T22:56:14.7300316Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96916
2023-01-11T22:56:14.7300681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7300911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7301296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7301468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7301831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7302002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7302371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7302556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7302787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7303034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7303263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7303501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7303886Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7304299Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7304558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8zaoyx3e
2023-01-11T22:56:14.7304830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8zaoyx3e/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7305086Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp34_4vpj1
2023-01-11T22:56:14.7305351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp34_4vpj1/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7305593Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7305828Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7305912Z ok (5.868s)
2023-01-11T22:56:14.7305949Z 
2023-01-11T22:56:14.7306199Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7306310Z Ran 1 test in 5.869s
2023-01-11T22:56:14.7306329Z 
2023-01-11T22:56:14.7306420Z OK
2023-01-11T22:56:14.7306439Z 
2023-01-11T22:56:14.7306561Z Generating XML reports...
2023-01-11T22:56:14.7307021Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225042.xml
2023-01-11T22:56:14.7307389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7307564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7308046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7308224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7308261Z 
2023-01-11T22:56:14.7308351Z Running tests...
2023-01-11T22:56:14.7308613Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7308922Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7309209Z   test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7309427Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97036
2023-01-11T22:56:14.7309644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97037
2023-01-11T22:56:14.7310010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7310172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7310551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7310740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7311102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7311275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7311647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7312074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7312327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7312563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7312652Z ok (21.288s)
2023-01-11T22:56:14.7312673Z 
2023-01-11T22:56:14.7312947Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7313057Z Ran 1 test in 21.288s
2023-01-11T22:56:14.7313077Z 
2023-01-11T22:56:14.7313169Z OK
2023-01-11T22:56:14.7313188Z 
2023-01-11T22:56:14.7313312Z Generating XML reports...
2023-01-11T22:56:14.7313771Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225050.xml
2023-01-11T22:56:14.7314138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7314313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7314670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7314865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7314888Z 
2023-01-11T22:56:14.7314998Z Running tests...
2023-01-11T22:56:14.7315259Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7315571Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7315754Z   test_no_grad (__main__.DistributedDataParallelTest)
2023-01-11T22:56:14.7316006Z Note: this test can be sped up by only running it on a CPU module ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7316224Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97157
2023-01-11T22:56:14.7316440Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97158
2023-01-11T22:56:14.7316790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7316963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7317489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7317681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7318043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7318219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7318590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7318773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7318985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7319232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7319464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7319702Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7320102Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7320497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7320757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyi5b89u8
2023-01-11T22:56:14.7321028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyi5b89u8/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7321283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_bb817r
2023-01-11T22:56:14.7321530Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_bb817r/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7321637Z ok (5.863s)
2023-01-11T22:56:14.7321660Z 
2023-01-11T22:56:14.7321926Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7322036Z Ran 1 test in 5.863s
2023-01-11T22:56:14.7322056Z 
2023-01-11T22:56:14.7322146Z OK
2023-01-11T22:56:14.7322165Z 
2023-01-11T22:56:14.7322288Z Generating XML reports...
2023-01-11T22:56:14.7322747Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225114.xml
2023-01-11T22:56:14.7323116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7323289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7323648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7323833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7323857Z 
2023-01-11T22:56:14.7323967Z Running tests...
2023-01-11T22:56:14.7324229Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7324540Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7324821Z   test_param_layout_mismatch_error (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7325040Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97274
2023-01-11T22:56:14.7325255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97275
2023-01-11T22:56:14.7325606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7325780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7326155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7326441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7326814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7326986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7327354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7327543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7327773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7328004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7328223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7328462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7328866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7329263Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7329522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvm0nesr7
2023-01-11T22:56:14.7329794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvm0nesr7/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7330048Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcaid4rmn
2023-01-11T22:56:14.7330301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcaid4rmn/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7330403Z ok (5.349s)
2023-01-11T22:56:14.7330424Z 
2023-01-11T22:56:14.7330688Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7330806Z Ran 1 test in 5.349s
2023-01-11T22:56:14.7330825Z 
2023-01-11T22:56:14.7330917Z OK
2023-01-11T22:56:14.7330935Z 
2023-01-11T22:56:14.7331058Z Generating XML reports...
2023-01-11T22:56:14.7331515Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225122.xml
2023-01-11T22:56:14.7331887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7332061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7332421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7332610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7332630Z 
2023-01-11T22:56:14.7332736Z Running tests...
2023-01-11T22:56:14.7332995Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7333318Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7333589Z   test_pass_default_pg (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7333807Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97391
2023-01-11T22:56:14.7334023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97392
2023-01-11T22:56:14.7334377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7334553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7334930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7335117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7335583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7335758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7336133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7336320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7336549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7336779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7337003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7337240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7337644Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7338042Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7338144Z ok (3.870s)
2023-01-11T22:56:14.7338164Z 
2023-01-11T22:56:14.7338427Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7338536Z Ran 1 test in 3.870s
2023-01-11T22:56:14.7338556Z 
2023-01-11T22:56:14.7338645Z OK
2023-01-11T22:56:14.7338665Z 
2023-01-11T22:56:14.7338769Z Generating XML reports...
2023-01-11T22:56:14.7339226Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225129.xml
2023-01-11T22:56:14.7339595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7339768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7340151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7340340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7340360Z 
2023-01-11T22:56:14.7340467Z Running tests...
2023-01-11T22:56:14.7340727Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7341021Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7341308Z   test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7341526Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97498
2023-01-11T22:56:14.7341742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97499
2023-01-11T22:56:14.7342107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7342287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7342665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7342853Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7343215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7343367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7343741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7343924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7344155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7344398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7344723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7344968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7345377Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7345930Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:56:14.7346378Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7346930Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:56:14.7347197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw4nmsbjk
2023-01-11T22:56:14.7347471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw4nmsbjk/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7347726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpktis7kfn
2023-01-11T22:56:14.7347996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpktis7kfn/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7348545Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7349092Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7349630Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:56:14.7350179Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:56:14.7350722Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7351252Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7352148Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:56:14.7352715Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:56:14.7353249Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7353782Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7354325Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:56:14.7354859Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:56:14.7355409Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7355943Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7356046Z ok (5.470s)
2023-01-11T22:56:14.7356067Z 
2023-01-11T22:56:14.7356353Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7356468Z Ran 1 test in 5.470s
2023-01-11T22:56:14.7356487Z 
2023-01-11T22:56:14.7356578Z OK
2023-01-11T22:56:14.7356597Z 
2023-01-11T22:56:14.7356721Z Generating XML reports...
2023-01-11T22:56:14.7357167Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225136.xml
2023-01-11T22:56:14.7357539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7357713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7358090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7358280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7358300Z 
2023-01-11T22:56:14.7358406Z Running tests...
2023-01-11T22:56:14.7358666Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7359142Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7359432Z   test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7359652Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97619
2023-01-11T22:56:14.7359868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97620
2023-01-11T22:56:14.7360239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7360412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7360789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7360979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7361349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7361519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7361873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7362058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7362288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7362534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7362761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7362998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7363406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7363954Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:56:14.7364351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7364896Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:56:14.7365163Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__plzwyc
2023-01-11T22:56:14.7365416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__plzwyc/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7365674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk1rk5kqz
2023-01-11T22:56:14.7365946Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk1rk5kqz/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7366491Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7367078Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7367661Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:56:14.7368202Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:56:14.7368743Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7369277Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7369814Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:56:14.7370355Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:56:14.7370887Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7371415Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7371963Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:56:14.7372496Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = True
2023-01-11T22:56:14.7373084Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7373662Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1000; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:56:14.7373765Z ok (5.478s)
2023-01-11T22:56:14.7373786Z 
2023-01-11T22:56:14.7374057Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7374169Z Ran 1 test in 5.478s
2023-01-11T22:56:14.7374189Z 
2023-01-11T22:56:14.7374278Z OK
2023-01-11T22:56:14.7374297Z 
2023-01-11T22:56:14.7374419Z Generating XML reports...
2023-01-11T22:56:14.7374886Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225143.xml
2023-01-11T22:56:14.7375244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7375417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7375795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7375984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7376004Z 
2023-01-11T22:56:14.7376111Z Running tests...
2023-01-11T22:56:14.7376373Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7376681Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7376961Z   test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7377168Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97740
2023-01-11T22:56:14.7377387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97741
2023-01-11T22:56:14.7377757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7377931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7378314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7378503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7378861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7379031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7379401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7379575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7379806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7380052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7380279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7380514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7380916Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7381311Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7381572Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpan9orpe6
2023-01-11T22:56:14.7381935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpan9orpe6/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7382178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp93tpzpk5
2023-01-11T22:56:14.7382447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp93tpzpk5/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7382683Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7382912Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7383013Z ok (6.284s)
2023-01-11T22:56:14.7383033Z 
2023-01-11T22:56:14.7383300Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7383410Z Ran 1 test in 6.284s
2023-01-11T22:56:14.7383430Z 
2023-01-11T22:56:14.7383521Z OK
2023-01-11T22:56:14.7383539Z 
2023-01-11T22:56:14.7383649Z Generating XML reports...
2023-01-11T22:56:14.7384112Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225151.xml
2023-01-11T22:56:14.7384483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7384657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7385034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7385223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7385244Z 
2023-01-11T22:56:14.7385350Z Running tests...
2023-01-11T22:56:14.7385613Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7385924Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7386195Z   test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7386419Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97861
2023-01-11T22:56:14.7386634Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97862
2023-01-11T22:56:14.7387004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7387176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7387551Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7387739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7388100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7388251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7388630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7388814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7389043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7389287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7389509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7389751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7390151Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7390545Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7390881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe0p8oetb
2023-01-11T22:56:14.7391161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe0p8oetb/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7391418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfoprfzfc
2023-01-11T22:56:14.7391685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfoprfzfc/_remote_module_non_scriptable.py
2023-01-11T22:56:14.7392329Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7392576Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:56:14.7392680Z ok (6.053s)
2023-01-11T22:56:14.7392700Z 
2023-01-11T22:56:14.7392977Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7393069Z Ran 1 test in 6.054s
2023-01-11T22:56:14.7393113Z 
2023-01-11T22:56:14.7393190Z OK
2023-01-11T22:56:14.7393209Z 
2023-01-11T22:56:14.7393334Z Generating XML reports...
2023-01-11T22:56:14.7393800Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225200.xml
2023-01-11T22:56:14.7394170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7394344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7394721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7394911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7394931Z 
2023-01-11T22:56:14.7395038Z Running tests...
2023-01-11T22:56:14.7395282Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7395594Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7395869Z   test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7396089Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97982
2023-01-11T22:56:14.7396303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97983
2023-01-11T22:56:14.7396514Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 97984
2023-01-11T22:56:14.7396886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7397063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7397436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7397608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7397977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7398148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7398516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7398703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7399059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7399228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7399603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7399772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7400002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7400371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7400601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:56:14.7400748Z skip: Need at least 3 CUDA devices (3.946s)
2023-01-11T22:56:14.7400769Z 
2023-01-11T22:56:14.7401080Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7401192Z Ran 1 test in 3.946s
2023-01-11T22:56:14.7401212Z 
2023-01-11T22:56:14.7401320Z OK (skipped=1)
2023-01-11T22:56:14.7401338Z 
2023-01-11T22:56:14.7401460Z Generating XML reports...
2023-01-11T22:56:14.7401879Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225208.xml
2023-01-11T22:56:14.7402250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7402428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7402809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7402997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7403016Z 
2023-01-11T22:56:14.7403123Z Running tests...
2023-01-11T22:56:14.7403382Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7403693Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7403943Z   test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7404161Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98119
2023-01-11T22:56:14.7404377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98120
2023-01-11T22:56:14.7404591Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 98121
2023-01-11T22:56:14.7404965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7405138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7405512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7405699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7406061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7406216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7406587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7406771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7407137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7407311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7407679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7407863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7408091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:56:14.7408298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7408524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7408669Z skip: Need at least 3 CUDA devices (3.934s)
2023-01-11T22:56:14.7408689Z 
2023-01-11T22:56:14.7408951Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7409154Z Ran 1 test in 3.934s
2023-01-11T22:56:14.7409175Z 
2023-01-11T22:56:14.7409285Z OK (skipped=1)
2023-01-11T22:56:14.7409303Z 
2023-01-11T22:56:14.7409426Z Generating XML reports...
2023-01-11T22:56:14.7409860Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225214.xml
2023-01-11T22:56:14.7410232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7410388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7410766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7410956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7410976Z 
2023-01-11T22:56:14.7411083Z Running tests...
2023-01-11T22:56:14.7411342Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7411661Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7411999Z   test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest) ... skip: Frequently times out see https://github.com/pytorch/pytorch/issues/58920 (0.000s)
2023-01-11T22:56:14.7412018Z 
2023-01-11T22:56:14.7412274Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7412366Z Ran 1 test in 0.001s
2023-01-11T22:56:14.7412403Z 
2023-01-11T22:56:14.7412490Z OK (skipped=1)
2023-01-11T22:56:14.7412509Z 
2023-01-11T22:56:14.7412631Z Generating XML reports...
2023-01-11T22:56:14.7413058Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225221.xml
2023-01-11T22:56:14.7413423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7413596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7413978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7414167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7414187Z 
2023-01-11T22:56:14.7414294Z Running tests...
2023-01-11T22:56:14.7414537Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7414845Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7415114Z   test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7415332Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98289
2023-01-11T22:56:14.7415547Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98290
2023-01-11T22:56:14.7415759Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 98291
2023-01-11T22:56:14.7416134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7416308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7416686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7416856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7417220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7417391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7417761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7417945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7418419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7418596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7418964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7419130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7419359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7419584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:56:14.7419806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7419953Z skip: Need at least 3 CUDA devices (3.911s)
2023-01-11T22:56:14.7419973Z 
2023-01-11T22:56:14.7420233Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7420348Z Ran 1 test in 3.911s
2023-01-11T22:56:14.7420371Z 
2023-01-11T22:56:14.7420476Z OK (skipped=1)
2023-01-11T22:56:14.7420495Z 
2023-01-11T22:56:14.7420617Z Generating XML reports...
2023-01-11T22:56:14.7421035Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225223.xml
2023-01-11T22:56:14.7421404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7421576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7421950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7422138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7422157Z 
2023-01-11T22:56:14.7422265Z Running tests...
2023-01-11T22:56:14.7422524Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7422845Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7423098Z   test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7423316Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98426
2023-01-11T22:56:14.7423529Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98427
2023-01-11T22:56:14.7423743Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 98428
2023-01-11T22:56:14.7424114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7424286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7424659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7424857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7425222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7425377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7425747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7425929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7426288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7426458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7426832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7427015Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7427345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7427562Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:56:14.7427787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7427886Z ok (3.958s)
2023-01-11T22:56:14.7427906Z 
2023-01-11T22:56:14.7428170Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7428279Z Ran 1 test in 3.959s
2023-01-11T22:56:14.7428299Z 
2023-01-11T22:56:14.7428390Z OK
2023-01-11T22:56:14.7428409Z 
2023-01-11T22:56:14.7428533Z Generating XML reports...
2023-01-11T22:56:14.7428966Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225229.xml
2023-01-11T22:56:14.7429328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7429493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7429869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7430055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7430075Z 
2023-01-11T22:56:14.7430181Z Running tests...
2023-01-11T22:56:14.7430439Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7430750Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7431016Z   test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7431233Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98563
2023-01-11T22:56:14.7431432Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98564
2023-01-11T22:56:14.7431656Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 98565
2023-01-11T22:56:14.7432288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7432473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7432855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7433044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7433403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7433574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7433926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7434113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7434481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7434652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7435019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7435205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7435435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7435655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:56:14.7435879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7435962Z ok (4.020s)
2023-01-11T22:56:14.7435982Z 
2023-01-11T22:56:14.7436244Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7436512Z Ran 1 test in 4.021s
2023-01-11T22:56:14.7436533Z 
2023-01-11T22:56:14.7436627Z OK
2023-01-11T22:56:14.7436647Z 
2023-01-11T22:56:14.7436770Z Generating XML reports...
2023-01-11T22:56:14.7437212Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225235.xml
2023-01-11T22:56:14.7437580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7437754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7438113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7438303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7438323Z 
2023-01-11T22:56:14.7438430Z Running tests...
2023-01-11T22:56:14.7438694Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7439013Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7439278Z   test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7439497Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98700
2023-01-11T22:56:14.7439710Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98701
2023-01-11T22:56:14.7439924Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 98702
2023-01-11T22:56:14.7440276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7440449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7440816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7440996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7441374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7441561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7441934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7442119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7442465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7442640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7443010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7443197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7443434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7443662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:56:14.7443887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7443988Z ok (3.928s)
2023-01-11T22:56:14.7444008Z 
2023-01-11T22:56:14.7444271Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7444365Z Ran 1 test in 3.928s
2023-01-11T22:56:14.7444384Z 
2023-01-11T22:56:14.7444476Z OK
2023-01-11T22:56:14.7444495Z 
2023-01-11T22:56:14.7444619Z Generating XML reports...
2023-01-11T22:56:14.7445051Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225242.xml
2023-01-11T22:56:14.7445414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7445684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7446067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7446253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7446273Z 
2023-01-11T22:56:14.7446363Z Running tests...
2023-01-11T22:56:14.7446622Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7446930Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7447197Z   test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest) ... skip: Test does not pass when run locally (0.001s)
2023-01-11T22:56:14.7447216Z 
2023-01-11T22:56:14.7447473Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7447583Z Ran 1 test in 0.001s
2023-01-11T22:56:14.7447605Z 
2023-01-11T22:56:14.7447711Z OK (skipped=1)
2023-01-11T22:56:14.7447733Z 
2023-01-11T22:56:14.7447854Z Generating XML reports...
2023-01-11T22:56:14.7448282Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225248.xml
2023-01-11T22:56:14.7448629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7448803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7449174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7449361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7449380Z 
2023-01-11T22:56:14.7449486Z Running tests...
2023-01-11T22:56:14.7449744Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7450055Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7450305Z   test_nccl_timeout (__main__.NcclErrorHandlingTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7450507Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98870
2023-01-11T22:56:14.7450722Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98871
2023-01-11T22:56:14.7450935Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 98872
2023-01-11T22:56:14.7451305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7451480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7451857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7452044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7452412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7452583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7452940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7453127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7453483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7453654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7454021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7454208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7454435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7454765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7454981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:56:14.7455131Z skip: Need at least 3 CUDA devices (4.045s)
2023-01-11T22:56:14.7455152Z 
2023-01-11T22:56:14.7455419Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7455530Z Ran 1 test in 4.045s
2023-01-11T22:56:14.7455549Z 
2023-01-11T22:56:14.7455656Z OK (skipped=1)
2023-01-11T22:56:14.7455674Z 
2023-01-11T22:56:14.7455798Z Generating XML reports...
2023-01-11T22:56:14.7456232Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225250.xml
2023-01-11T22:56:14.7456597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7456774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7457134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7457324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7457343Z 
2023-01-11T22:56:14.7457448Z Running tests...
2023-01-11T22:56:14.7457710Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7458020Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7458352Z   test_all_to_all_single (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7458570Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99007
2023-01-11T22:56:14.7458937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7459099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7459476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7459663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7459890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7460134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7460535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:56:14.7460637Z ok (4.640s)
2023-01-11T22:56:14.7460656Z 
2023-01-11T22:56:14.7460916Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7461028Z Ran 1 test in 4.640s
2023-01-11T22:56:14.7461047Z 
2023-01-11T22:56:14.7461120Z OK
2023-01-11T22:56:14.7461143Z 
2023-01-11T22:56:14.7461268Z Generating XML reports...
2023-01-11T22:56:14.7461821Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111225256.xml
2023-01-11T22:56:14.7462189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7462361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7462737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7462927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7462949Z 
2023-01-11T22:56:14.7463056Z Running tests...
2023-01-11T22:56:14.7463315Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7463608Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7464042Z   test_allgather_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7464269Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99082
2023-01-11T22:56:14.7464638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7464809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7465184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7465374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7465601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7465829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7466237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:56:14.7466340Z ok (4.642s)
2023-01-11T22:56:14.7466360Z 
2023-01-11T22:56:14.7466626Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7466738Z Ran 1 test in 4.642s
2023-01-11T22:56:14.7466757Z 
2023-01-11T22:56:14.7466850Z OK
2023-01-11T22:56:14.7466869Z 
2023-01-11T22:56:14.7466991Z Generating XML reports...
2023-01-11T22:56:14.7467540Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111225303.xml
2023-01-11T22:56:14.7467904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7468062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7468439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7468633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7468654Z 
2023-01-11T22:56:14.7468762Z Running tests...
2023-01-11T22:56:14.7469022Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7469333Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7469677Z   test_allreduce_coalesced (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7469897Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99157
2023-01-11T22:56:14.7470248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7470422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7470802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7470994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7471226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7471469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7472122Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:56:14.7472897Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:56:14.7473007Z   warnings.warn(
2023-01-11T22:56:14.7473091Z ok (4.731s)
2023-01-11T22:56:14.7473130Z 
2023-01-11T22:56:14.7473527Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7473648Z Ran 1 test in 4.731s
2023-01-11T22:56:14.7473667Z 
2023-01-11T22:56:14.7473758Z OK
2023-01-11T22:56:14.7473778Z 
2023-01-11T22:56:14.7473901Z Generating XML reports...
2023-01-11T22:56:14.7474455Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111225310.xml
2023-01-11T22:56:14.7474824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7474998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7475376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7475547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7475584Z 
2023-01-11T22:56:14.7475674Z Running tests...
2023-01-11T22:56:14.7475939Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7476253Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7476584Z   test_collectives (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7476802Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99232
2023-01-11T22:56:14.7477172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7477347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7477725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7477897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7478125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7478377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7478783Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:56:14.7478884Z ok (4.532s)
2023-01-11T22:56:14.7478903Z 
2023-01-11T22:56:14.7479164Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7479275Z Ran 1 test in 4.532s
2023-01-11T22:56:14.7479295Z 
2023-01-11T22:56:14.7479387Z OK
2023-01-11T22:56:14.7479406Z 
2023-01-11T22:56:14.7479514Z Generating XML reports...
2023-01-11T22:56:14.7480062Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111225317.xml
2023-01-11T22:56:14.7480430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7480610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7480987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7481176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7481196Z 
2023-01-11T22:56:14.7481304Z Running tests...
2023-01-11T22:56:14.7481562Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7481875Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7482199Z   test_reduce_scatter_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7482417Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99307
2023-01-11T22:56:14.7482781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7483068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7483455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7483643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7483871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7484116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7484501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:56:14.7484605Z ok (4.766s)
2023-01-11T22:56:14.7484625Z 
2023-01-11T22:56:14.7484884Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7484995Z Ran 1 test in 4.766s
2023-01-11T22:56:14.7485018Z 
2023-01-11T22:56:14.7485109Z OK
2023-01-11T22:56:14.7485128Z 
2023-01-11T22:56:14.7485253Z Generating XML reports...
2023-01-11T22:56:14.7485804Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111225324.xml
2023-01-11T22:56:14.7486171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7486344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7486702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7486889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7486909Z 
2023-01-11T22:56:14.7487016Z Running tests...
2023-01-11T22:56:14.7487275Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7487585Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7487854Z   test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest) ... skip: GPUs are available, skipping test (0.001s)
2023-01-11T22:56:14.7487874Z 
2023-01-11T22:56:14.7488132Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7488241Z Ran 1 test in 0.001s
2023-01-11T22:56:14.7488260Z 
2023-01-11T22:56:14.7488366Z OK (skipped=1)
2023-01-11T22:56:14.7488386Z 
2023-01-11T22:56:14.7488491Z Generating XML reports...
2023-01-11T22:56:14.7488951Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20230111225331.xml
2023-01-11T22:56:14.7489318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7489490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7489864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7490058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7490077Z 
2023-01-11T22:56:14.7490186Z Running tests...
2023-01-11T22:56:14.7490447Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7490740Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7491001Z   test_allgather_base_basics (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7491217Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99415
2023-01-11T22:56:14.7491433Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99416
2023-01-11T22:56:14.7491802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7491975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7492468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7492664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7493030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7493185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7493558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7493744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7493978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7494225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7494455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7494698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7495099Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7495477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7495579Z ok (4.736s)
2023-01-11T22:56:14.7495599Z 
2023-01-11T22:56:14.7495861Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7495971Z Ran 1 test in 4.736s
2023-01-11T22:56:14.7495991Z 
2023-01-11T22:56:14.7496083Z OK
2023-01-11T22:56:14.7496102Z 
2023-01-11T22:56:14.7496224Z Generating XML reports...
2023-01-11T22:56:14.7496657Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225333.xml
2023-01-11T22:56:14.7497031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7497205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7497562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7497750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7497769Z 
2023-01-11T22:56:14.7497877Z Running tests...
2023-01-11T22:56:14.7498136Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7498447Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7498700Z   test_allgather_base_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7498918Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99524
2023-01-11T22:56:14.7499141Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99525
2023-01-11T22:56:14.7499492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7499668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7500046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7500233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7500592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7500766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7501191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7501522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7501757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7501985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7502216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7502455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7502858Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7503254Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7503355Z ok (5.372s)
2023-01-11T22:56:14.7503376Z 
2023-01-11T22:56:14.7503637Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7503755Z Ran 1 test in 5.372s
2023-01-11T22:56:14.7503774Z 
2023-01-11T22:56:14.7503865Z OK
2023-01-11T22:56:14.7503884Z 
2023-01-11T22:56:14.7503990Z Generating XML reports...
2023-01-11T22:56:14.7504422Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225340.xml
2023-01-11T22:56:14.7504786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7504958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7505335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7505523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7505542Z 
2023-01-11T22:56:14.7505653Z Running tests...
2023-01-11T22:56:14.7505916Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7506218Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7506462Z   test_allgather_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7506679Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99641
2023-01-11T22:56:14.7506894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99642
2023-01-11T22:56:14.7507262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7507434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7507809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7507994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7508357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7508513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7508888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7509073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7509301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7509546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7509768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7510006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7510409Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7510891Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7510999Z ok (5.340s)
2023-01-11T22:56:14.7511019Z 
2023-01-11T22:56:14.7511286Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7511397Z Ran 1 test in 5.340s
2023-01-11T22:56:14.7511418Z 
2023-01-11T22:56:14.7511512Z OK
2023-01-11T22:56:14.7511531Z 
2023-01-11T22:56:14.7511655Z Generating XML reports...
2023-01-11T22:56:14.7512333Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225348.xml
2023-01-11T22:56:14.7512720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7512893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7513256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7513449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7513469Z 
2023-01-11T22:56:14.7513579Z Running tests...
2023-01-11T22:56:14.7513842Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7514153Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7514399Z   test_allreduce_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7514615Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99758
2023-01-11T22:56:14.7514831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99759
2023-01-11T22:56:14.7515181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7515354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7515734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7515923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7516288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7516458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7516830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7517016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7517244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7517471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7517704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7517949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7518353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7518749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7518851Z ok (5.391s)
2023-01-11T22:56:14.7518871Z 
2023-01-11T22:56:14.7519132Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7519241Z Ran 1 test in 5.391s
2023-01-11T22:56:14.7519261Z 
2023-01-11T22:56:14.7519333Z OK
2023-01-11T22:56:14.7519369Z 
2023-01-11T22:56:14.7519474Z Generating XML reports...
2023-01-11T22:56:14.7519905Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225356.xml
2023-01-11T22:56:14.7520423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7520603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7520979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7521168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7521187Z 
2023-01-11T22:56:14.7521295Z Running tests...
2023-01-11T22:56:14.7521555Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7521848Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7522082Z   test_barrier (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7522299Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99875
2023-01-11T22:56:14.7522525Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99876
2023-01-11T22:56:14.7522892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7523064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7523435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7523623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7523968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7524141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7524510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7524705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7524932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7525176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7525398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7525633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7526031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7526415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7526518Z ok (5.350s)
2023-01-11T22:56:14.7526538Z 
2023-01-11T22:56:14.7526799Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7526917Z Ran 1 test in 5.350s
2023-01-11T22:56:14.7526936Z 
2023-01-11T22:56:14.7527027Z OK
2023-01-11T22:56:14.7527046Z 
2023-01-11T22:56:14.7527168Z Generating XML reports...
2023-01-11T22:56:14.7527597Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225403.xml
2023-01-11T22:56:14.7527965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7528139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7528501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7528689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7528708Z 
2023-01-11T22:56:14.7528815Z Running tests...
2023-01-11T22:56:14.7529078Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7529490Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7529740Z   test_broadcast_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7529957Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99992
2023-01-11T22:56:14.7530172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99993
2023-01-11T22:56:14.7530525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7530699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7531072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7531259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7531620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7531797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7532165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7532349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7532560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7532805Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7533028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7533270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7533673Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7534076Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7534178Z ok (5.378s)
2023-01-11T22:56:14.7534199Z 
2023-01-11T22:56:14.7534460Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7534570Z Ran 1 test in 5.378s
2023-01-11T22:56:14.7534590Z 
2023-01-11T22:56:14.7534662Z OK
2023-01-11T22:56:14.7534681Z 
2023-01-11T22:56:14.7534806Z Generating XML reports...
2023-01-11T22:56:14.7535239Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225411.xml
2023-01-11T22:56:14.7535606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7535782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7536160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7536355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7536375Z 
2023-01-11T22:56:14.7536483Z Running tests...
2023-01-11T22:56:14.7536743Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7537036Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7537278Z   test_empty_tensors (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7537496Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100109
2023-01-11T22:56:14.7537712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100110
2023-01-11T22:56:14.7538079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7538309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7538729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7538921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7539264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7539436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7539803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7539988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7540215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7540461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7540694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7540930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7541332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7541710Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7541813Z ok (5.483s)
2023-01-11T22:56:14.7541833Z 
2023-01-11T22:56:14.7542093Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7542204Z Ran 1 test in 5.484s
2023-01-11T22:56:14.7542224Z 
2023-01-11T22:56:14.7542314Z OK
2023-01-11T22:56:14.7542333Z 
2023-01-11T22:56:14.7542456Z Generating XML reports...
2023-01-11T22:56:14.7542888Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225419.xml
2023-01-11T22:56:14.7543259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7543415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7543795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7543985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7544004Z 
2023-01-11T22:56:14.7544112Z Running tests...
2023-01-11T22:56:14.7544372Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7544684Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7544930Z   test_gather_checks (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7545149Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100226
2023-01-11T22:56:14.7545372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100227
2023-01-11T22:56:14.7545726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7545900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7546276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7546464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7546824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7546995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7547366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7547673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7547890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7548135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7548357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7548594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7548999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7549394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7549496Z ok (4.767s)
2023-01-11T22:56:14.7549517Z 
2023-01-11T22:56:14.7549787Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7549899Z Ran 1 test in 4.767s
2023-01-11T22:56:14.7549918Z 
2023-01-11T22:56:14.7549991Z OK
2023-01-11T22:56:14.7550009Z 
2023-01-11T22:56:14.7550133Z Generating XML reports...
2023-01-11T22:56:14.7550564Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225427.xml
2023-01-11T22:56:14.7550931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7551103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7551477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7551669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7551689Z 
2023-01-11T22:56:14.7551797Z Running tests...
2023-01-11T22:56:14.7552323Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7552627Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7552869Z   test_gather_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7553092Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100335
2023-01-11T22:56:14.7553307Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100336
2023-01-11T22:56:14.7553675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7553849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7554226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7554418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7554772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7554945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7555318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7555503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7555732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7555975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7556201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7556443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7556925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7557372Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7557474Z ok (5.356s)
2023-01-11T22:56:14.7557497Z 
2023-01-11T22:56:14.7557760Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7557870Z Ran 1 test in 5.356s
2023-01-11T22:56:14.7557890Z 
2023-01-11T22:56:14.7557981Z OK
2023-01-11T22:56:14.7557999Z 
2023-01-11T22:56:14.7558122Z Generating XML reports...
2023-01-11T22:56:14.7558554Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225434.xml
2023-01-11T22:56:14.7558917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7559074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7559459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7559648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7559667Z 
2023-01-11T22:56:14.7559775Z Running tests...
2023-01-11T22:56:14.7560036Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7560349Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7560593Z   test_gather_stress (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7560814Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100456
2023-01-11T22:56:14.7561031Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100457
2023-01-11T22:56:14.7561383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7561563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7561940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7562129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7562487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7562657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7563030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7563215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7563425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7563673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7563907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7564144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7564545Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7564939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7565040Z ok (9.874s)
2023-01-11T22:56:14.7565061Z 
2023-01-11T22:56:14.7565322Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7565431Z Ran 1 test in 9.874s
2023-01-11T22:56:14.7565450Z 
2023-01-11T22:56:14.7565522Z OK
2023-01-11T22:56:14.7565541Z 
2023-01-11T22:56:14.7565665Z Generating XML reports...
2023-01-11T22:56:14.7566148Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225441.xml
2023-01-11T22:56:14.7566570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7566742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7567120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7567310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7567329Z 
2023-01-11T22:56:14.7567438Z Running tests...
2023-01-11T22:56:14.7567682Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7567991Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7568251Z   test_nccl_dist_backend_error (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7568479Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100577
2023-01-11T22:56:14.7568696Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100578
2023-01-11T22:56:14.7569062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7569235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7569614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7569802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7570145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7570316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7570695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7570888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7571119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7571364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7571589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7571825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7572209Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7572604Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7572706Z ok (4.858s)
2023-01-11T22:56:14.7572729Z 
2023-01-11T22:56:14.7572997Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7573108Z Ran 1 test in 4.858s
2023-01-11T22:56:14.7573128Z 
2023-01-11T22:56:14.7573220Z OK
2023-01-11T22:56:14.7573239Z 
2023-01-11T22:56:14.7573362Z Generating XML reports...
2023-01-11T22:56:14.7573790Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225454.xml
2023-01-11T22:56:14.7574155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7574310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7574685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7574876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7574895Z 
2023-01-11T22:56:14.7575062Z Running tests...
2023-01-11T22:56:14.7575365Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7575683Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7575921Z   test_reduce_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7576141Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100689
2023-01-11T22:56:14.7576338Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100690
2023-01-11T22:56:14.7576706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7576879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7577255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7577443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7577810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7577982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7578357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7578545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7578754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7579000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7579221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7579463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7579868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7580264Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7580366Z ok (5.377s)
2023-01-11T22:56:14.7580386Z 
2023-01-11T22:56:14.7580649Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7580759Z Ran 1 test in 5.377s
2023-01-11T22:56:14.7580779Z 
2023-01-11T22:56:14.7580851Z OK
2023-01-11T22:56:14.7580870Z 
2023-01-11T22:56:14.7580995Z Generating XML reports...
2023-01-11T22:56:14.7581424Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225501.xml
2023-01-11T22:56:14.7581792Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7581967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7582349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7582539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7582558Z 
2023-01-11T22:56:14.7582666Z Running tests...
2023-01-11T22:56:14.7582909Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7583223Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7583488Z   test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7583707Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100806
2023-01-11T22:56:14.7583923Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100807
2023-01-11T22:56:14.7584288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7584559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7584943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7585131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7585474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7585644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7586015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7586200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7586429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7586680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7586909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7587147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7587533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7587931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7588033Z ok (4.779s)
2023-01-11T22:56:14.7588053Z 
2023-01-11T22:56:14.7588320Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7588430Z Ran 1 test in 4.780s
2023-01-11T22:56:14.7588449Z 
2023-01-11T22:56:14.7588541Z OK
2023-01-11T22:56:14.7588560Z 
2023-01-11T22:56:14.7588683Z Generating XML reports...
2023-01-11T22:56:14.7589123Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225508.xml
2023-01-11T22:56:14.7589489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7589644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7590016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7590204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7590223Z 
2023-01-11T22:56:14.7590331Z Running tests...
2023-01-11T22:56:14.7590592Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7590904Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7591168Z   test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7591394Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100915
2023-01-11T22:56:14.7591594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100916
2023-01-11T22:56:14.7592203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7592392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7592771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7592959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7593321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7593493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7594011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7594206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7594417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7594660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7594882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7595117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7595521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7595915Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7596020Z ok (5.397s)
2023-01-11T22:56:14.7596043Z 
2023-01-11T22:56:14.7596305Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7596416Z Ran 1 test in 5.397s
2023-01-11T22:56:14.7596436Z 
2023-01-11T22:56:14.7596508Z OK
2023-01-11T22:56:14.7596526Z 
2023-01-11T22:56:14.7596653Z Generating XML reports...
2023-01-11T22:56:14.7597085Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225515.xml
2023-01-11T22:56:14.7597450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7597623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7598001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7598190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7598213Z 
2023-01-11T22:56:14.7598321Z Running tests...
2023-01-11T22:56:14.7598566Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7598880Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7599134Z   test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7599350Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101032
2023-01-11T22:56:14.7599567Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101033
2023-01-11T22:56:14.7599934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7600107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7600480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7600676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7601071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7601248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7601619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7601803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7602034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7602280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7602501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7602743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7603234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7603638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7603739Z ok (5.375s)
2023-01-11T22:56:14.7603759Z 
2023-01-11T22:56:14.7604018Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7604128Z Ran 1 test in 5.375s
2023-01-11T22:56:14.7604148Z 
2023-01-11T22:56:14.7604241Z OK
2023-01-11T22:56:14.7604260Z 
2023-01-11T22:56:14.7604383Z Generating XML reports...
2023-01-11T22:56:14.7604815Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225523.xml
2023-01-11T22:56:14.7605181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7605344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7605721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7605909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7605929Z 
2023-01-11T22:56:14.7606039Z Running tests...
2023-01-11T22:56:14.7606300Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7606612Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7606859Z   test_scatter_checks (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7607078Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101149
2023-01-11T22:56:14.7607277Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101150
2023-01-11T22:56:14.7607651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7607828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7608203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7608391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7608751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7608923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7609292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7609477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7609687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7609942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7610170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7610408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7610812Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7611212Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7611312Z ok (4.767s)
2023-01-11T22:56:14.7611332Z 
2023-01-11T22:56:14.7611594Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7611951Z Ran 1 test in 4.768s
2023-01-11T22:56:14.7612119Z 
2023-01-11T22:56:14.7612211Z OK
2023-01-11T22:56:14.7612408Z 
2023-01-11T22:56:14.7612535Z Generating XML reports...
2023-01-11T22:56:14.7613162Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225531.xml
2023-01-11T22:56:14.7613878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7614329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7614895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7615345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7615578Z 
2023-01-11T22:56:14.7615689Z Running tests...
2023-01-11T22:56:14.7616091Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7616620Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7617102Z   test_scatter_ops (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7617575Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101258
2023-01-11T22:56:14.7618031Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101259
2023-01-11T22:56:14.7618625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7619083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7619659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7620133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7620691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7621131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7621710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7622178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7622597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7623138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7623627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7624115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7624760Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7625454Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7625859Z ok (5.354s)
2023-01-11T22:56:14.7626008Z 
2023-01-11T22:56:14.7626257Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7626599Z Ran 1 test in 5.354s
2023-01-11T22:56:14.7626764Z 
2023-01-11T22:56:14.7626863Z OK
2023-01-11T22:56:14.7626996Z 
2023-01-11T22:56:14.7627126Z Generating XML reports...
2023-01-11T22:56:14.7627701Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225538.xml
2023-01-11T22:56:14.7628401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7628852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7629422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7629872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7630163Z 
2023-01-11T22:56:14.7630314Z Running tests...
2023-01-11T22:56:14.7630723Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7631230Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7631723Z   test_scatter_stress (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7632440Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101379
2023-01-11T22:56:14.7632893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101380
2023-01-11T22:56:14.7633488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7633941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7634510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7634976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7635557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7636005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7636579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7637046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7637495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7637989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7638478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7638960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7641874Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7642629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7643027Z ok (9.840s)
2023-01-11T22:56:14.7643156Z 
2023-01-11T22:56:14.7643425Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7643751Z Ran 1 test in 9.840s
2023-01-11T22:56:14.7643912Z 
2023-01-11T22:56:14.7644005Z OK
2023-01-11T22:56:14.7644137Z 
2023-01-11T22:56:14.7644242Z Generating XML reports...
2023-01-11T22:56:14.7644836Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225546.xml
2023-01-11T22:56:14.7645535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7646001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7646557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7647027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7647262Z 
2023-01-11T22:56:14.7647371Z Running tests...
2023-01-11T22:56:14.7647756Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7648317Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7648828Z   test_send_recv (__main__.ProcessGroupNCCLTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7649309Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101500
2023-01-11T22:56:14.7649762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101501
2023-01-11T22:56:14.7650600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7651060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7651639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7652113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7652671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7653116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7653683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7654129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7654567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:56:14.7655074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:56:14.7655573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:56:14.7656040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7656753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7657450Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:56:14.7657869Z ok (4.841s)
2023-01-11T22:56:14.7658022Z 
2023-01-11T22:56:14.7658289Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7658599Z Ran 1 test in 4.841s
2023-01-11T22:56:14.7658759Z 
2023-01-11T22:56:14.7658860Z OK
2023-01-11T22:56:14.7658995Z 
2023-01-11T22:56:14.7659123Z Generating XML reports...
2023-01-11T22:56:14.7659715Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225558.xml
2023-01-11T22:56:14.7660423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7660879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7661458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7661905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7662133Z 
2023-01-11T22:56:14.7662245Z Running tests...
2023-01-11T22:56:14.7662642Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7663165Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7663638Z   test_common_errors (__main__.RendezvousEnvTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7664129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7664791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:56:14.7665308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7665956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:56:14.7666478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7667117Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:56:14.7667632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7668393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:56:14.7668787Z ok (1.677s)
2023-01-11T22:56:14.7668933Z 
2023-01-11T22:56:14.7669203Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7669509Z Ran 1 test in 1.677s
2023-01-11T22:56:14.7669666Z 
2023-01-11T22:56:14.7669759Z OK
2023-01-11T22:56:14.7669889Z 
2023-01-11T22:56:14.7670016Z Generating XML reports...
2023-01-11T22:56:14.7670565Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20230111225605.xml
2023-01-11T22:56:14.7671245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:56:14.7671693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:56:14.7672783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:56:14.7673241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:56:14.7673472Z 
2023-01-11T22:56:14.7673580Z Running tests...
2023-01-11T22:56:14.7673987Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7674500Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T22:56:14.7674984Z   test_default_store_timeout_nccl (__main__.TimeoutTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:56:14.7675479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7676132Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:56:14.7676652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:56:14.7677305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:56:14.7677694Z ok (4.680s)
2023-01-11T22:56:14.7677838Z 
2023-01-11T22:56:14.7678103Z ----------------------------------------------------------------------
2023-01-11T22:56:14.7678408Z Ran 1 test in 4.680s
2023-01-11T22:56:14.7678567Z 
2023-01-11T22:56:14.7678659Z OK
2023-01-11T22:56:14.7678790Z 
2023-01-11T22:56:14.7678912Z Generating XML reports...
2023-01-11T22:56:14.7679442Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20230111225609.xml
2023-01-11T22:56:14.7679763Z 
2023-01-11T22:56:14.7680175Z ##[endgroup]
2023-01-11T22:56:14.7680740Z FINISHED PRINTING LOG FILE of distributed/test_c10d_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_nccl_hu8yk7hw)
2023-01-11T22:56:14.7681070Z 
2023-01-11T22:56:14.9288983Z 
2023-01-11T22:56:14.9289628Z real	93m15.629s
2023-01-11T22:56:14.9290064Z user	155m5.667s
2023-01-11T22:56:14.9290297Z sys	85m59.601s
2023-01-11T22:56:14.9290544Z + assert_git_not_dirty
2023-01-11T22:56:14.9291067Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 != *rocm* ]]
2023-01-11T22:56:14.9291488Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 != *xla* ]]
2023-01-11T22:56:14.9293398Z ++ git status --porcelain
2023-01-11T22:56:16.5294010Z + git_status=
2023-01-11T22:56:16.5294429Z + [[ -n '' ]]
2023-01-11T22:56:16.5294834Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *cuda* ]]
2023-01-11T22:56:16.5295137Z + [[ 2 == 1 ]]
2023-01-11T22:56:16.5295341Z + [[ 2 == 1 ]]
2023-01-11T22:56:16.5367434Z Prepare all required actions
2023-01-11T22:56:16.5367880Z Getting action download info
2023-01-11T22:56:16.7485808Z ##[group]Run ./.github/actions/get-workflow-job-id
2023-01-11T22:56:16.7486094Z with:
2023-01-11T22:56:16.7486580Z   github-token: ***
2023-01-11T22:56:16.7486833Z env:
2023-01-11T22:56:16.7487063Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:56:16.7487337Z   GPU_FLAG: --gpus all
2023-01-11T22:56:16.7487894Z   DOCKER_CONTAINER_ID: c5cb3f34fdb745cbab81ead9bbd8979c6cd118fca3d50c4cb582659c15cdaef3
2023-01-11T22:56:16.7488252Z ##[endgroup]
2023-01-11T22:56:16.7522221Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2023-01-11T22:56:16.7522541Z with:
2023-01-11T22:56:16.7522768Z   shell: bash
2023-01-11T22:56:16.7522997Z   timeout_minutes: 10
2023-01-11T22:56:16.7523247Z   max_attempts: 5
2023-01-11T22:56:16.7523500Z   retry_wait_seconds: 30
2023-01-11T22:56:16.7524037Z   command: set -eux
python3 -m pip install requests==2.26.0
GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}")
echo "job-id=${GHA_WORKFLOW_JOB_ID}" >> "${GITHUB_OUTPUT}"

2023-01-11T22:56:16.7524561Z   polling_interval_seconds: 1
2023-01-11T22:56:16.7524819Z   warning_on_retry: true
2023-01-11T22:56:16.7525084Z   continue_on_error: false
2023-01-11T22:56:16.7525336Z env:
2023-01-11T22:56:16.7525557Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:56:16.7525832Z   GPU_FLAG: --gpus all
2023-01-11T22:56:16.7526224Z   DOCKER_CONTAINER_ID: c5cb3f34fdb745cbab81ead9bbd8979c6cd118fca3d50c4cb582659c15cdaef3
2023-01-11T22:56:16.7526755Z   GITHUB_TOKEN: ***
2023-01-11T22:56:16.7526986Z ##[endgroup]
2023-01-11T22:56:16.8217194Z + python3 -m pip install requests==2.26.0
2023-01-11T22:56:17.1190840Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T22:56:17.1421563Z Requirement already satisfied: requests==2.26.0 in /home/ec2-user/.local/lib/python3.7/site-packages (2.26.0)
2023-01-11T22:56:17.1612199Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2.0.12)
2023-01-11T22:56:17.1639564Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2022.12.7)
2023-01-11T22:56:17.1651939Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (3.4)
2023-01-11T22:56:17.1668305Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (1.26.14)
2023-01-11T22:56:17.4402842Z ++ python3 .github/scripts/get_workflow_job_id.py 3896346758 i-0ac6ae9883f837720
2023-01-11T22:56:21.6953203Z + GHA_WORKFLOW_JOB_ID=10589560479
2023-01-11T22:56:21.6954227Z + echo job-id=10589560479
2023-01-11T22:56:21.8229298Z Command completed after 1 attempt(s).
2023-01-11T22:56:21.8387920Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2023-01-11T22:56:21.8388314Z kill "$MONITOR_SCRIPT_PID"
2023-01-11T22:56:21.8403073Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T22:56:21.8403396Z env:
2023-01-11T22:56:21.8403659Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:56:21.8403913Z   GPU_FLAG: --gpus all
2023-01-11T22:56:21.8404314Z   DOCKER_CONTAINER_ID: c5cb3f34fdb745cbab81ead9bbd8979c6cd118fca3d50c4cb582659c15cdaef3
2023-01-11T22:56:21.8404734Z   MONITOR_SCRIPT_PID: 54575
2023-01-11T22:56:21.8404983Z ##[endgroup]
2023-01-11T22:56:21.8535489Z Prepare all required actions
2023-01-11T22:56:21.8535906Z Getting action download info
2023-01-11T22:56:22.0733866Z Download action repository 'actions/upload-artifact@v3' (SHA:0b7f8abb1508181956e8e162db84b466c27e18ce)
2023-01-11T22:56:22.2709258Z ##[group]Run ./.github/actions/upload-test-artifacts
2023-01-11T22:56:22.2709563Z with:
2023-01-11T22:56:22.2709901Z   file-suffix: test-distributed-2-3-linux.8xlarge.nvidia.gpu_10589560479
2023-01-11T22:56:22.2710256Z env:
2023-01-11T22:56:22.2710500Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:56:22.2710752Z   GPU_FLAG: --gpus all
2023-01-11T22:56:22.2711137Z   DOCKER_CONTAINER_ID: c5cb3f34fdb745cbab81ead9bbd8979c6cd118fca3d50c4cb582659c15cdaef3
2023-01-11T22:56:22.2711517Z ##[endgroup]
2023-01-11T22:56:22.2743937Z ##[group]Run # Remove any previous test jsons if they exist
2023-01-11T22:56:22.2744301Z # Remove any previous test jsons if they exist
2023-01-11T22:56:22.2744771Z rm -f test-jsons-*.zip
2023-01-11T22:56:22.2745153Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2023-01-11T22:56:22.2757829Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T22:56:22.2758137Z env:
2023-01-11T22:56:22.2758387Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:56:22.2758637Z   GPU_FLAG: --gpus all
2023-01-11T22:56:22.2759029Z   DOCKER_CONTAINER_ID: c5cb3f34fdb745cbab81ead9bbd8979c6cd118fca3d50c4cb582659c15cdaef3
2023-01-11T22:56:22.2759531Z   FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_10589560479
2023-01-11T22:56:22.2759871Z ##[endgroup]
2023-01-11T22:56:22.2886019Z   adding: test/allowlist_for_publicAPI.json (deflated 78%)
2023-01-11T22:56:22.2920661Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2023-01-11T22:56:22.2928028Z   adding: test/profiler/profiler_utils_mock_events.json (deflated 87%)
2023-01-11T22:56:22.2929759Z   adding: test/.pytorch-slow-tests.json (deflated 77%)
2023-01-11T22:56:22.2935640Z   adding: test/.pytorch-disabled-tests.json (deflated 84%)
2023-01-11T22:56:22.2961026Z ##[group]Run # Remove any previous test reports if they exist
2023-01-11T22:56:22.2961400Z # Remove any previous test reports if they exist
2023-01-11T22:56:22.2961727Z rm -f test-reports-*.zip
2023-01-11T22:56:22.2962082Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv'
2023-01-11T22:56:22.2974323Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T22:56:22.2974622Z env:
2023-01-11T22:56:22.2974863Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:56:22.2975112Z   GPU_FLAG: --gpus all
2023-01-11T22:56:22.2975496Z   DOCKER_CONTAINER_ID: c5cb3f34fdb745cbab81ead9bbd8979c6cd118fca3d50c4cb582659c15cdaef3
2023-01-11T22:56:22.2975990Z   FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_10589560479
2023-01-11T22:56:22.2976327Z ##[endgroup]
2023-01-11T22:56:22.3139172Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_2d_model_state_checkpoint/TEST-Test2dModelStateCheckpoint-20230111212306.xml (deflated 45%)
2023-01-11T22:56:22.3140091Z   adding: test/test-reports/python-unittest/distributed._tensor.test_device_mesh/TEST-DeviceMeshCollectiveTest-20230111212312.xml (deflated 88%)
2023-01-11T22:56:22.3140870Z   adding: test/test-reports/python-unittest/distributed._tensor.test_device_mesh/TEST-DeviceMeshTest-20230111212312.xml (deflated 72%)
2023-01-11T22:56:22.3145386Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state/TEST-TestFSDPOptimState-20230111212405.xml (deflated 93%)
2023-01-11T22:56:22.3146179Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_dedup_tensors/TEST-TestDedupTensor-20230111221839.xml (deflated 40%)
2023-01-11T22:56:22.3146964Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_nested_dict/TEST-TestFlattening-20230111221843.xml (deflated 54%)
2023-01-11T22:56:22.3147904Z   adding: test/test-reports/python-unittest/distributed._composable.test_contract/TEST-TestContract-20230111221847.xml (deflated 68%)
2023-01-11T22:56:22.3148718Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestGetSubmoduleToStates-20230111221851.xml (deflated 44%)
2023-01-11T22:56:22.3149460Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_utils/TEST-TestUtils-20230111221851.xml (deflated 69%)
2023-01-11T22:56:22.3150188Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_fx/TEST-TestSymbolicTracing-20230111221855.xml (deflated 43%)
2023-01-11T22:56:22.3150944Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20230111221859.xml (deflated 62%)
2023-01-11T22:56:22.3151678Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20230111221859.xml (deflated 69%)
2023-01-11T22:56:22.3152872Z   adding: test/test-reports/python-unittest/distributed.rpc.test_share_memory/TEST-TestRPCPickler-20230111221905.xml (deflated 42%)
2023-01-11T22:56:22.3153976Z   adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20230111221914.xml (deflated 83%)
2023-01-11T22:56:22.3154788Z   adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCUDA-20230111221914.xml (deflated 90%)
2023-01-11T22:56:22.3155611Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20230111221921.xml (deflated 41%)
2023-01-11T22:56:22.3156421Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_wrapping/TEST-TestMultipleWrapping-20230111221929.xml (deflated 47%)
2023-01-11T22:56:22.3157251Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20230111221937.xml (deflated 59%)
2023-01-11T22:56:22.3158074Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20230111221945.xml (deflated 61%)
2023-01-11T22:56:22.3158917Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20230111221954.xml (deflated 59%)
2023-01-11T22:56:22.3159774Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestCustomShardingSpec-20230111222005.xml (deflated 66%)
2023-01-11T22:56:22.3160606Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_spec.test_sharding_spec/TEST-TestShardingSpec-20230111222005.xml (deflated 79%)
2023-01-11T22:56:22.3161444Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20230111222016.xml (deflated 69%)
2023-01-11T22:56:22.3162363Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20230111222027.xml (deflated 74%)
2023-01-11T22:56:22.3163295Z   adding: test/test-reports/python-unittest/distributed.tensor.parallel.test_tp_examples/TEST-DistTensorParallelExampleTest-20230111222041.xml (deflated 65%)
2023-01-11T22:56:22.3164140Z   adding: test/test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorOps-20230111222056.xml (deflated 67%)
2023-01-11T22:56:22.3164924Z   adding: test/test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorReshard-20230111222056.xml (deflated 60%)
2023-01-11T22:56:22.3165747Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan/TEST-TestShardingPlan-20230111222112.xml (deflated 75%)
2023-01-11T22:56:22.3166564Z   adding: test/test-reports/python-unittest/distributed._composable.test_replicate/TEST-ReplicateStateDictTest-20230111222128.xml (deflated 60%)
2023-01-11T22:56:22.3167341Z   adding: test/test-reports/python-unittest/distributed._composable.test_replicate/TEST-ReplicateTest-20230111222128.xml (deflated 62%)
2023-01-11T22:56:22.3168228Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_tp_integration/TEST-TestTPFSDPIntegration-20230111222144.xml (deflated 80%)
2023-01-11T22:56:22.3169015Z   adding: test/test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestDistributed-20230111222205.xml (deflated 80%)
2023-01-11T22:56:22.3169800Z   adding: test/test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestDistributedMultiProc-20230111222205.xml (deflated 75%)
2023-01-11T22:56:22.3170637Z   adding: test/test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestFakeDistributedSingleProc-20230111222205.xml (deflated 59%)
2023-01-11T22:56:22.3171507Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedCheckpointing-20230111222229.xml (deflated 56%)
2023-01-11T22:56:22.3172354Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedFailure-20230111222229.xml (deflated 77%)
2023-01-11T22:56:22.3173270Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-RunProcResultsTest-20230111222254.xml (deflated 55%)
2023-01-11T22:56:22.3174140Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesListTest-20230111222254.xml (deflated 81%)
2023-01-11T22:56:22.3175005Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StartProcessesTest-20230111222254.xml (deflated 80%)
2023-01-11T22:56:22.3175803Z   adding: test/test-reports/python-unittest/distributed.elastic.multiprocessing.api_test/TEST-StdTest-20230111222254.xml (deflated 64%)
2023-01-11T22:56:22.3176574Z   adding: test/test-reports/python-unittest/distributed._tensor.test_matrix_ops/TEST-DistMatrixOpsTest-20230111222323.xml (deflated 76%)
2023-01-11T22:56:22.3177360Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20230111222355.xml (deflated 85%)
2023-01-11T22:56:22.3178140Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_comm/TEST-TestCommunication-20230111222429.xml (deflated 91%)
2023-01-11T22:56:22.3178976Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20230111222509.xml (deflated 44%)
2023-01-11T22:56:22.3179906Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20230111222512.xml (deflated 43%)
2023-01-11T22:56:22.3180829Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-DistributedDataParallelSingleProcessTest-20230111222516.xml (deflated 43%)
2023-01-11T22:56:22.3181711Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111222520.xml (deflated 42%)
2023-01-11T22:56:22.3182548Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111222528.xml (deflated 42%)
2023-01-11T22:56:22.3183385Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111222535.xml (deflated 42%)
2023-01-11T22:56:22.3184243Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111222542.xml (deflated 42%)
2023-01-11T22:56:22.3185077Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111222550.xml (deflated 42%)
2023-01-11T22:56:22.3185901Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111222557.xml (deflated 42%)
2023-01-11T22:56:22.3186723Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111222604.xml (deflated 42%)
2023-01-11T22:56:22.3187617Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_gloo/TEST-TestDistributedNNFunctionsGloo-20230111222612.xml (deflated 42%)
2023-01-11T22:56:22.3188375Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-CommTest-20230111222620.xml (deflated 38%)
2023-01-11T22:56:22.3189123Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20230111222626.xml (deflated 41%)
2023-01-11T22:56:22.3189920Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20230111222630.xml (deflated 40%)
2023-01-11T22:56:22.3190726Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20230111222634.xml (deflated 40%)
2023-01-11T22:56:22.3191532Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ComputeBucketAssignmentTest-20230111222638.xml (deflated 41%)
2023-01-11T22:56:22.3192932Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20230111222642.xml (deflated 42%)
2023-01-11T22:56:22.3193889Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20230111222648.xml (deflated 41%)
2023-01-11T22:56:22.3194738Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20230111222657.xml (deflated 41%)
2023-01-11T22:56:22.3195596Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-PythonProcessGroupExtensionTest-20230111222703.xml (deflated 41%)
2023-01-11T22:56:22.3196353Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20230111222711.xml (deflated 39%)
2023-01-11T22:56:22.3197045Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20230111222715.xml (deflated 39%)
2023-01-11T22:56:22.3197711Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20230111222719.xml (deflated 39%)
2023-01-11T22:56:22.3198406Z   adding: test/test-reports/python-unittest/distributed.test_c10d_common/TEST-ReduceOpTest-20230111222723.xml (deflated 39%)
2023-01-11T22:56:22.3199100Z   adding: test/test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorTest-20230111222727.xml (deflated 75%)
2023-01-11T22:56:22.3199818Z   adding: test/test-reports/python-unittest/distributed._tensor.test_dtensor/TEST-DTensorMeshTest-20230111222727.xml (deflated 74%)
2023-01-11T22:56:22.3200583Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222808.xml (deflated 41%)
2023-01-11T22:56:22.3201458Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222815.xml (deflated 40%)
2023-01-11T22:56:22.3202269Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222821.xml (deflated 41%)
2023-01-11T22:56:22.3203088Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222828.xml (deflated 41%)
2023-01-11T22:56:22.3203870Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222834.xml (deflated 40%)
2023-01-11T22:56:22.3204686Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222840.xml (deflated 41%)
2023-01-11T22:56:22.3205494Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222847.xml (deflated 41%)
2023-01-11T22:56:22.3206303Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222853.xml (deflated 40%)
2023-01-11T22:56:22.3207086Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupGlooWrapperTest-20230111222900.xml (deflated 40%)
2023-01-11T22:56:22.3207976Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111222906.xml (deflated 40%)
2023-01-11T22:56:22.3208809Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111222913.xml (deflated 39%)
2023-01-11T22:56:22.3209617Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111222920.xml (deflated 39%)
2023-01-11T22:56:22.3210402Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111222928.xml (deflated 40%)
2023-01-11T22:56:22.3211219Z   adding: test/test-reports/python-unittest/distributed.test_pg_wrapper/TEST-ProcessGroupNCCLWrapperTest-20230111222936.xml (deflated 39%)
2023-01-11T22:56:22.3211971Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20230111222944.xml (deflated 81%)
2023-01-11T22:56:22.3212745Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20230111222944.xml (deflated 89%)
2023-01-11T22:56:22.3213554Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestCreateTensorFromParams-20230111223111.xml (deflated 43%)
2023-01-11T22:56:22.3214423Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorMetadata-20230111223111.xml (deflated 44%)
2023-01-11T22:56:22.3215257Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestLocalTensor-20230111223111.xml (deflated 60%)
2023-01-11T22:56:22.3216065Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestModuleHookApi-20230111223111.xml (deflated 59%)
2023-01-11T22:56:22.3216882Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardMetadata-20230111223111.xml (deflated 58%)
2023-01-11T22:56:22.3217681Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardParameter-20230111223111.xml (deflated 60%)
2023-01-11T22:56:22.3218492Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardTensor-20230111223111.xml (deflated 61%)
2023-01-11T22:56:22.3219327Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorChunked-20230111223111.xml (deflated 88%)
2023-01-11T22:56:22.3220198Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorCustomOps-20230111223111.xml (deflated 69%)
2023-01-11T22:56:22.3221066Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorEnumerable-20230111223111.xml (deflated 86%)
2023-01-11T22:56:22.3221983Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalShards-20230111223111.xml (deflated 83%)
2023-01-11T22:56:22.3222916Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor/TEST-TestShardedTensorFromLocalTensor-20230111223111.xml (deflated 62%)
2023-01-11T22:56:22.3223776Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20230111223341.xml (deflated 96%)
2023-01-11T22:56:22.3224489Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224023.xml (deflated 39%)
2023-01-11T22:56:22.3225132Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224031.xml (deflated 38%)
2023-01-11T22:56:22.3225794Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224038.xml (deflated 39%)
2023-01-11T22:56:22.3226450Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224045.xml (deflated 38%)
2023-01-11T22:56:22.3227172Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224052.xml (deflated 38%)
2023-01-11T22:56:22.3227826Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224059.xml (deflated 39%)
2023-01-11T22:56:22.3228483Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224105.xml (deflated 39%)
2023-01-11T22:56:22.3229137Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224111.xml (deflated 39%)
2023-01-11T22:56:22.3229790Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224118.xml (deflated 38%)
2023-01-11T22:56:22.3230424Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224125.xml (deflated 38%)
2023-01-11T22:56:22.3231079Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224133.xml (deflated 37%)
2023-01-11T22:56:22.3231791Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224141.xml (deflated 37%)
2023-01-11T22:56:22.3232879Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224147.xml (deflated 38%)
2023-01-11T22:56:22.3233519Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224155.xml (deflated 38%)
2023-01-11T22:56:22.3234168Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224202.xml (deflated 39%)
2023-01-11T22:56:22.3234823Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224208.xml (deflated 38%)
2023-01-11T22:56:22.3235473Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224215.xml (deflated 37%)
2023-01-11T22:56:22.3236114Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224223.xml (deflated 38%)
2023-01-11T22:56:22.3236768Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111224230.xml (deflated 38%)
2023-01-11T22:56:22.3237437Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224237.xml (deflated 38%)
2023-01-11T22:56:22.3238106Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224245.xml (deflated 38%)
2023-01-11T22:56:22.3238820Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224253.xml (deflated 38%)
2023-01-11T22:56:22.3239508Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224301.xml (deflated 38%)
2023-01-11T22:56:22.3240184Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224308.xml (deflated 38%)
2023-01-11T22:56:22.3240873Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224316.xml (deflated 38%)
2023-01-11T22:56:22.3241535Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224324.xml (deflated 38%)
2023-01-11T22:56:22.3242217Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224331.xml (deflated 38%)
2023-01-11T22:56:22.3242905Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224339.xml (deflated 38%)
2023-01-11T22:56:22.3243587Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111224347.xml (deflated 38%)
2023-01-11T22:56:22.3244327Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224354.xml (deflated 41%)
2023-01-11T22:56:22.3245154Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224403.xml (deflated 41%)
2023-01-11T22:56:22.3246090Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224411.xml (deflated 41%)
2023-01-11T22:56:22.3246930Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224419.xml (deflated 41%)
2023-01-11T22:56:22.3247711Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224427.xml (deflated 41%)
2023-01-11T22:56:22.3248509Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224435.xml (deflated 42%)
2023-01-11T22:56:22.3249316Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224443.xml (deflated 41%)
2023-01-11T22:56:22.3250128Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224450.xml (deflated 41%)
2023-01-11T22:56:22.3251179Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224458.xml (deflated 42%)
2023-01-11T22:56:22.3251978Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224506.xml (deflated 45%)
2023-01-11T22:56:22.3252780Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224513.xml (deflated 45%)
2023-01-11T22:56:22.3253588Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224521.xml (deflated 43%)
2023-01-11T22:56:22.3254368Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224529.xml (deflated 43%)
2023-01-11T22:56:22.3255166Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224537.xml (deflated 45%)
2023-01-11T22:56:22.3255974Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224545.xml (deflated 45%)
2023-01-11T22:56:22.3256773Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224552.xml (deflated 47%)
2023-01-11T22:56:22.3257560Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224600.xml (deflated 47%)
2023-01-11T22:56:22.3258368Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224608.xml (deflated 44%)
2023-01-11T22:56:22.3259157Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224616.xml (deflated 46%)
2023-01-11T22:56:22.3259954Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224624.xml (deflated 46%)
2023-01-11T22:56:22.3260801Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224631.xml (deflated 43%)
2023-01-11T22:56:22.3261586Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224639.xml (deflated 43%)
2023-01-11T22:56:22.3262387Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224647.xml (deflated 42%)
2023-01-11T22:56:22.3263190Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224655.xml (deflated 41%)
2023-01-11T22:56:22.3263990Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224703.xml (deflated 42%)
2023-01-11T22:56:22.3264772Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224710.xml (deflated 45%)
2023-01-11T22:56:22.3265632Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224718.xml (deflated 44%)
2023-01-11T22:56:22.3266442Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224726.xml (deflated 41%)
2023-01-11T22:56:22.3267244Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224732.xml (deflated 44%)
2023-01-11T22:56:22.3268021Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224740.xml (deflated 41%)
2023-01-11T22:56:22.3268825Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224748.xml (deflated 41%)
2023-01-11T22:56:22.3269618Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224755.xml (deflated 41%)
2023-01-11T22:56:22.3270420Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224802.xml (deflated 41%)
2023-01-11T22:56:22.3271260Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224810.xml (deflated 42%)
2023-01-11T22:56:22.3272467Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224818.xml (deflated 42%)
2023-01-11T22:56:22.3273443Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224822.xml (deflated 42%)
2023-01-11T22:56:22.3274244Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224826.xml (deflated 42%)
2023-01-11T22:56:22.3275025Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224830.xml (deflated 42%)
2023-01-11T22:56:22.3275826Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224834.xml (deflated 42%)
2023-01-11T22:56:22.3276629Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224838.xml (deflated 41%)
2023-01-11T22:56:22.3277372Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20230111225609.xml (deflated 39%)
2023-01-11T22:56:22.3278108Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224842.xml (deflated 42%)
2023-01-11T22:56:22.3278916Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224850.xml (deflated 41%)
2023-01-11T22:56:22.3279713Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224858.xml (deflated 41%)
2023-01-11T22:56:22.3280514Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224905.xml (deflated 41%)
2023-01-11T22:56:22.3281330Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224913.xml (deflated 40%)
2023-01-11T22:56:22.3282114Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224923.xml (deflated 42%)
2023-01-11T22:56:22.3282906Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224929.xml (deflated 41%)
2023-01-11T22:56:22.3283705Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224935.xml (deflated 42%)
2023-01-11T22:56:22.3284507Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224943.xml (deflated 42%)
2023-01-11T22:56:22.3285294Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111224951.xml (deflated 41%)
2023-01-11T22:56:22.3286188Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225000.xml (deflated 42%)
2023-01-11T22:56:22.3287007Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225008.xml (deflated 42%)
2023-01-11T22:56:22.3287806Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225014.xml (deflated 42%)
2023-01-11T22:56:22.3288589Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225020.xml (deflated 41%)
2023-01-11T22:56:22.3289389Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225027.xml (deflated 43%)
2023-01-11T22:56:22.3290187Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225034.xml (deflated 42%)
2023-01-11T22:56:22.3291106Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225042.xml (deflated 42%)
2023-01-11T22:56:22.3291910Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225050.xml (deflated 43%)
2023-01-11T22:56:22.3292691Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225114.xml (deflated 44%)
2023-01-11T22:56:22.3293481Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225122.xml (deflated 42%)
2023-01-11T22:56:22.3294278Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225129.xml (deflated 42%)
2023-01-11T22:56:22.3295080Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225136.xml (deflated 41%)
2023-01-11T22:56:22.3295878Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225143.xml (deflated 40%)
2023-01-11T22:56:22.3296677Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225151.xml (deflated 41%)
2023-01-11T22:56:22.3297476Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111225200.xml (deflated 41%)
2023-01-11T22:56:22.3298257Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225208.xml (deflated 41%)
2023-01-11T22:56:22.3299011Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225214.xml (deflated 41%)
2023-01-11T22:56:22.3299776Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225221.xml (deflated 42%)
2023-01-11T22:56:22.3300547Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225223.xml (deflated 42%)
2023-01-11T22:56:22.3301362Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225229.xml (deflated 41%)
2023-01-11T22:56:22.3302100Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225235.xml (deflated 41%)
2023-01-11T22:56:22.3302862Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225242.xml (deflated 41%)
2023-01-11T22:56:22.3303616Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225248.xml (deflated 42%)
2023-01-11T22:56:22.3304370Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111225250.xml (deflated 41%)
2023-01-11T22:56:22.3305241Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111225256.xml (deflated 43%)
2023-01-11T22:56:22.3306289Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111225303.xml (deflated 42%)
2023-01-11T22:56:22.3307276Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111225310.xml (deflated 42%)
2023-01-11T22:56:22.3308256Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111225317.xml (deflated 43%)
2023-01-11T22:56:22.3309216Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111225324.xml (deflated 42%)
2023-01-11T22:56:22.3310111Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20230111225331.xml (deflated 41%)
2023-01-11T22:56:22.3310958Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225333.xml (deflated 38%)
2023-01-11T22:56:22.3311720Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225340.xml (deflated 38%)
2023-01-11T22:56:22.3312881Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225348.xml (deflated 38%)
2023-01-11T22:56:22.3313651Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225356.xml (deflated 39%)
2023-01-11T22:56:22.3314413Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225403.xml (deflated 39%)
2023-01-11T22:56:22.3315170Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225411.xml (deflated 39%)
2023-01-11T22:56:22.3315904Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225419.xml (deflated 39%)
2023-01-11T22:56:22.3316668Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225427.xml (deflated 38%)
2023-01-11T22:56:22.3317427Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225434.xml (deflated 39%)
2023-01-11T22:56:22.3318189Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225441.xml (deflated 39%)
2023-01-11T22:56:22.3318923Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225454.xml (deflated 39%)
2023-01-11T22:56:22.3319679Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225501.xml (deflated 39%)
2023-01-11T22:56:22.3320429Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225508.xml (deflated 39%)
2023-01-11T22:56:22.3321188Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225515.xml (deflated 39%)
2023-01-11T22:56:22.3321914Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225523.xml (deflated 39%)
2023-01-11T22:56:22.3322666Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225531.xml (deflated 38%)
2023-01-11T22:56:22.3323416Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225538.xml (deflated 38%)
2023-01-11T22:56:22.3324168Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225546.xml (deflated 39%)
2023-01-11T22:56:22.3324901Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111225558.xml (deflated 38%)
2023-01-11T22:56:22.3325642Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20230111225605.xml (deflated 40%)
2023-01-11T22:56:22.3326573Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212755.xml (deflated 43%)
2023-01-11T22:56:22.3327497Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212757.xml (deflated 44%)
2023-01-11T22:56:22.3328394Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212759.xml (deflated 42%)
2023-01-11T22:56:22.3329283Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212807.xml (deflated 43%)
2023-01-11T22:56:22.3330170Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212814.xml (deflated 42%)
2023-01-11T22:56:22.3331144Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212822.xml (deflated 42%)
2023-01-11T22:56:22.3332034Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212832.xml (deflated 43%)
2023-01-11T22:56:22.3332909Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212834.xml (deflated 44%)
2023-01-11T22:56:22.3333802Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212836.xml (deflated 42%)
2023-01-11T22:56:22.3334685Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212844.xml (deflated 42%)
2023-01-11T22:56:22.3335566Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212851.xml (deflated 42%)
2023-01-11T22:56:22.3336442Z   adding: test/test-reports/dist-nccl/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212859.xml (deflated 42%)
2023-01-11T22:56:22.3337345Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212909.xml (deflated 41%)
2023-01-11T22:56:22.3338143Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212917.xml (deflated 42%)
2023-01-11T22:56:22.3338931Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212920.xml (deflated 42%)
2023-01-11T22:56:22.3339725Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212926.xml (deflated 43%)
2023-01-11T22:56:22.3340515Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212930.xml (deflated 42%)
2023-01-11T22:56:22.3341296Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212933.xml (deflated 42%)
2023-01-11T22:56:22.3342087Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212935.xml (deflated 40%)
2023-01-11T22:56:22.3342873Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212944.xml (deflated 40%)
2023-01-11T22:56:22.3343655Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212953.xml (deflated 40%)
2023-01-11T22:56:22.3344420Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213001.xml (deflated 39%)
2023-01-11T22:56:22.3345204Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213009.xml (deflated 40%)
2023-01-11T22:56:22.3346049Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213018.xml (deflated 40%)
2023-01-11T22:56:22.3346838Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213027.xml (deflated 40%)
2023-01-11T22:56:22.3347605Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213035.xml (deflated 42%)
2023-01-11T22:56:22.3348386Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213039.xml (deflated 41%)
2023-01-11T22:56:22.3349166Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213046.xml (deflated 42%)
2023-01-11T22:56:22.3349943Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213050.xml (deflated 42%)
2023-01-11T22:56:22.3350774Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213059.xml (deflated 42%)
2023-01-11T22:56:22.3351558Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213101.xml (deflated 46%)
2023-01-11T22:56:22.3353051Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213110.xml (deflated 47%)
2023-01-11T22:56:22.3353841Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213119.xml (deflated 48%)
2023-01-11T22:56:22.3354600Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213127.xml (deflated 46%)
2023-01-11T22:56:22.3355387Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213136.xml (deflated 41%)
2023-01-11T22:56:22.3356172Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213139.xml (deflated 44%)
2023-01-11T22:56:22.3356958Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213141.xml (deflated 44%)
2023-01-11T22:56:22.3357730Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213143.xml (deflated 44%)
2023-01-11T22:56:22.3358511Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213146.xml (deflated 44%)
2023-01-11T22:56:22.3359290Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213148.xml (deflated 43%)
2023-01-11T22:56:22.3360066Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213151.xml (deflated 42%)
2023-01-11T22:56:22.3360824Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213153.xml (deflated 41%)
2023-01-11T22:56:22.3361611Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213202.xml (deflated 41%)
2023-01-11T22:56:22.3362389Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213211.xml (deflated 41%)
2023-01-11T22:56:22.3363165Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213213.xml (deflated 41%)
2023-01-11T22:56:22.3363930Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213216.xml (deflated 40%)
2023-01-11T22:56:22.3364704Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213225.xml (deflated 40%)
2023-01-11T22:56:22.3365484Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213234.xml (deflated 42%)
2023-01-11T22:56:22.3366360Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213238.xml (deflated 42%)
2023-01-11T22:56:22.3367140Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213242.xml (deflated 41%)
2023-01-11T22:56:22.3367918Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213250.xml (deflated 40%)
2023-01-11T22:56:22.3368696Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213259.xml (deflated 41%)
2023-01-11T22:56:22.3369475Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213308.xml (deflated 41%)
2023-01-11T22:56:22.3370236Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213310.xml (deflated 41%)
2023-01-11T22:56:22.3371095Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213312.xml (deflated 41%)
2023-01-11T22:56:22.3371873Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213315.xml (deflated 41%)
2023-01-11T22:56:22.3372651Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213317.xml (deflated 41%)
2023-01-11T22:56:22.3373410Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213320.xml (deflated 41%)
2023-01-11T22:56:22.3374192Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213322.xml (deflated 41%)
2023-01-11T22:56:22.3374974Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213324.xml (deflated 41%)
2023-01-11T22:56:22.3375749Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213327.xml (deflated 41%)
2023-01-11T22:56:22.3376508Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213329.xml (deflated 41%)
2023-01-11T22:56:22.3377288Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213332.xml (deflated 41%)
2023-01-11T22:56:22.3378064Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213334.xml (deflated 41%)
2023-01-11T22:56:22.3378846Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213336.xml (deflated 41%)
2023-01-11T22:56:22.3379605Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213339.xml (deflated 41%)
2023-01-11T22:56:22.3380396Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213341.xml (deflated 41%)
2023-01-11T22:56:22.3381168Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213344.xml (deflated 41%)
2023-01-11T22:56:22.3381943Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213346.xml (deflated 41%)
2023-01-11T22:56:22.3382719Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213348.xml (deflated 41%)
2023-01-11T22:56:22.3383479Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213351.xml (deflated 41%)
2023-01-11T22:56:22.3384254Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213353.xml (deflated 41%)
2023-01-11T22:56:22.3385098Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213356.xml (deflated 41%)
2023-01-11T22:56:22.3385898Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213358.xml (deflated 41%)
2023-01-11T22:56:22.3386653Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213400.xml (deflated 42%)
2023-01-11T22:56:22.3387428Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213403.xml (deflated 41%)
2023-01-11T22:56:22.3388209Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213405.xml (deflated 43%)
2023-01-11T22:56:22.3388984Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213407.xml (deflated 43%)
2023-01-11T22:56:22.3389756Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213410.xml (deflated 42%)
2023-01-11T22:56:22.3390608Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213412.xml (deflated 41%)
2023-01-11T22:56:22.3391395Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213420.xml (deflated 42%)
2023-01-11T22:56:22.3392602Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213423.xml (deflated 41%)
2023-01-11T22:56:22.3393390Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213425.xml (deflated 41%)
2023-01-11T22:56:22.3394171Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213427.xml (deflated 40%)
2023-01-11T22:56:22.3394947Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213436.xml (deflated 40%)
2023-01-11T22:56:22.3395745Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213444.xml (deflated 41%)
2023-01-11T22:56:22.3396517Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213452.xml (deflated 42%)
2023-01-11T22:56:22.3397300Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213455.xml (deflated 42%)
2023-01-11T22:56:22.3398088Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213457.xml (deflated 41%)
2023-01-11T22:56:22.3398867Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213505.xml (deflated 41%)
2023-01-11T22:56:22.3399635Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213514.xml (deflated 43%)
2023-01-11T22:56:22.3400429Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213516.xml (deflated 41%)
2023-01-11T22:56:22.3401307Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213525.xml (deflated 43%)
2023-01-11T22:56:22.3402088Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213527.xml (deflated 42%)
2023-01-11T22:56:22.3402870Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213534.xml (deflated 42%)
2023-01-11T22:56:22.3403624Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213536.xml (deflated 42%)
2023-01-11T22:56:22.3404409Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213539.xml (deflated 42%)
2023-01-11T22:56:22.3405284Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213543.xml (deflated 41%)
2023-01-11T22:56:22.3406076Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213552.xml (deflated 42%)
2023-01-11T22:56:22.3406838Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213554.xml (deflated 42%)
2023-01-11T22:56:22.3407625Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213558.xml (deflated 42%)
2023-01-11T22:56:22.3408403Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213601.xml (deflated 42%)
2023-01-11T22:56:22.3409181Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213608.xml (deflated 42%)
2023-01-11T22:56:22.3410023Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213610.xml (deflated 42%)
2023-01-11T22:56:22.3410801Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213613.xml (deflated 41%)
2023-01-11T22:56:22.3411577Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213621.xml (deflated 41%)
2023-01-11T22:56:22.3412353Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213629.xml (deflated 42%)
2023-01-11T22:56:22.3413115Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213632.xml (deflated 40%)
2023-01-11T22:56:22.3413899Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213640.xml (deflated 42%)
2023-01-11T22:56:22.3414680Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213642.xml (deflated 42%)
2023-01-11T22:56:22.3415461Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213649.xml (deflated 41%)
2023-01-11T22:56:22.3416243Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213657.xml (deflated 42%)
2023-01-11T22:56:22.3417004Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213704.xml (deflated 42%)
2023-01-11T22:56:22.3417779Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213706.xml (deflated 42%)
2023-01-11T22:56:22.3418559Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213709.xml (deflated 40%)
2023-01-11T22:56:22.3419335Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213718.xml (deflated 42%)
2023-01-11T22:56:22.3420102Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213720.xml (deflated 42%)
2023-01-11T22:56:22.3420880Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213727.xml (deflated 42%)
2023-01-11T22:56:22.3421655Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213729.xml (deflated 42%)
2023-01-11T22:56:22.3422431Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213736.xml (deflated 42%)
2023-01-11T22:56:22.3423195Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213738.xml (deflated 42%)
2023-01-11T22:56:22.3423973Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213741.xml (deflated 42%)
2023-01-11T22:56:22.3424809Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213743.xml (deflated 41%)
2023-01-11T22:56:22.3425596Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213746.xml (deflated 40%)
2023-01-11T22:56:22.3426358Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213748.xml (deflated 41%)
2023-01-11T22:56:22.3427136Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213755.xml (deflated 41%)
2023-01-11T22:56:22.3427913Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213803.xml (deflated 41%)
2023-01-11T22:56:22.3428697Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213810.xml (deflated 41%)
2023-01-11T22:56:22.3429522Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213817.xml (deflated 41%)
2023-01-11T22:56:22.3430295Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213824.xml (deflated 40%)
2023-01-11T22:56:22.3431076Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213832.xml (deflated 41%)
2023-01-11T22:56:22.3432129Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213840.xml (deflated 41%)
2023-01-11T22:56:22.3433022Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213847.xml (deflated 41%)
2023-01-11T22:56:22.3433807Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213849.xml (deflated 42%)
2023-01-11T22:56:22.3434595Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213854.xml (deflated 41%)
2023-01-11T22:56:22.3435372Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213856.xml (deflated 42%)
2023-01-11T22:56:22.3436131Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213858.xml (deflated 43%)
2023-01-11T22:56:22.3436906Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213901.xml (deflated 42%)
2023-01-11T22:56:22.3437683Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213905.xml (deflated 42%)
2023-01-11T22:56:22.3438461Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213909.xml (deflated 42%)
2023-01-11T22:56:22.3439229Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213913.xml (deflated 40%)
2023-01-11T22:56:22.3440002Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213923.xml (deflated 40%)
2023-01-11T22:56:22.3440781Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213932.xml (deflated 40%)
2023-01-11T22:56:22.3441558Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213941.xml (deflated 40%)
2023-01-11T22:56:22.3442320Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213949.xml (deflated 40%)
2023-01-11T22:56:22.3443101Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213957.xml (deflated 42%)
2023-01-11T22:56:22.3443968Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214001.xml (deflated 42%)
2023-01-11T22:56:22.3444828Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214005.xml (deflated 40%)
2023-01-11T22:56:22.3445588Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214013.xml (deflated 40%)
2023-01-11T22:56:22.3446369Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214022.xml (deflated 41%)
2023-01-11T22:56:22.3447147Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214030.xml (deflated 40%)
2023-01-11T22:56:22.3447922Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214038.xml (deflated 42%)
2023-01-11T22:56:22.3448679Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214043.xml (deflated 42%)
2023-01-11T22:56:22.3449553Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214045.xml (deflated 42%)
2023-01-11T22:56:22.3450327Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214049.xml (deflated 41%)
2023-01-11T22:56:22.3451106Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214058.xml (deflated 42%)
2023-01-11T22:56:22.3451867Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214102.xml (deflated 42%)
2023-01-11T22:56:22.3452653Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214106.xml (deflated 40%)
2023-01-11T22:56:22.3453433Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214115.xml (deflated 40%)
2023-01-11T22:56:22.3454407Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214125.xml (deflated 42%)
2023-01-11T22:56:22.3455170Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214129.xml (deflated 41%)
2023-01-11T22:56:22.3456000Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214138.xml (deflated 42%)
2023-01-11T22:56:22.3456773Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214140.xml (deflated 42%)
2023-01-11T22:56:22.3457548Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214143.xml (deflated 42%)
2023-01-11T22:56:22.3458328Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214145.xml (deflated 41%)
2023-01-11T22:56:22.3459097Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214147.xml (deflated 41%)
2023-01-11T22:56:22.3459878Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214150.xml (deflated 41%)
2023-01-11T22:56:22.3460654Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214152.xml (deflated 41%)
2023-01-11T22:56:22.3461429Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214155.xml (deflated 41%)
2023-01-11T22:56:22.3462191Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214157.xml (deflated 41%)
2023-01-11T22:56:22.3462969Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214159.xml (deflated 42%)
2023-01-11T22:56:22.3463816Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214202.xml (deflated 42%)
2023-01-11T22:56:22.3464608Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214204.xml (deflated 42%)
2023-01-11T22:56:22.3465364Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214207.xml (deflated 42%)
2023-01-11T22:56:22.3466138Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214211.xml (deflated 41%)
2023-01-11T22:56:22.3466914Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214219.xml (deflated 41%)
2023-01-11T22:56:22.3467689Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214228.xml (deflated 41%)
2023-01-11T22:56:22.3468511Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214230.xml (deflated 41%)
2023-01-11T22:56:22.3469289Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214239.xml (deflated 40%)
2023-01-11T22:56:22.3470063Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214247.xml (deflated 40%)
2023-01-11T22:56:22.3470846Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214305.xml (deflated 41%)
2023-01-11T22:56:22.3471602Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214313.xml (deflated 41%)
2023-01-11T22:56:22.3472769Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214322.xml (deflated 41%)
2023-01-11T22:56:22.3473554Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214331.xml (deflated 40%)
2023-01-11T22:56:22.3474334Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214340.xml (deflated 42%)
2023-01-11T22:56:22.3475092Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214344.xml (deflated 42%)
2023-01-11T22:56:22.3475866Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214348.xml (deflated 42%)
2023-01-11T22:56:22.3476643Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214357.xml (deflated 40%)
2023-01-11T22:56:22.3477417Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214405.xml (deflated 42%)
2023-01-11T22:56:22.3478175Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214409.xml (deflated 40%)
2023-01-11T22:56:22.3478958Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214417.xml (deflated 42%)
2023-01-11T22:56:22.3479737Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214422.xml (deflated 41%)
2023-01-11T22:56:22.3480508Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214431.xml (deflated 41%)
2023-01-11T22:56:22.3481268Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214439.xml (deflated 40%)
2023-01-11T22:56:22.3482048Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214447.xml (deflated 42%)
2023-01-11T22:56:22.3482821Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214451.xml (deflated 42%)
2023-01-11T22:56:22.3483685Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214455.xml (deflated 42%)
2023-01-11T22:56:22.3484459Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214459.xml (deflated 40%)
2023-01-11T22:56:22.3485237Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214508.xml (deflated 41%)
2023-01-11T22:56:22.3486008Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214516.xml (deflated 40%)
2023-01-11T22:56:22.3486864Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214523.xml (deflated 40%)
2023-01-11T22:56:22.3487639Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214530.xml (deflated 42%)
2023-01-11T22:56:22.3488494Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214534.xml (deflated 42%)
2023-01-11T22:56:22.3489253Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214538.xml (deflated 40%)
2023-01-11T22:56:22.3490035Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214545.xml (deflated 41%)
2023-01-11T22:56:22.3490808Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214547.xml (deflated 41%)
2023-01-11T22:56:22.3491584Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214550.xml (deflated 40%)
2023-01-11T22:56:22.3492365Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214559.xml (deflated 41%)
2023-01-11T22:56:22.3493134Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214601.xml (deflated 41%)
2023-01-11T22:56:22.3493912Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214604.xml (deflated 40%)
2023-01-11T22:56:22.3494690Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214612.xml (deflated 42%)
2023-01-11T22:56:22.3495467Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214616.xml (deflated 41%)
2023-01-11T22:56:22.3496227Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214623.xml (deflated 40%)
2023-01-11T22:56:22.3497008Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214631.xml (deflated 41%)
2023-01-11T22:56:22.3497793Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214638.xml (deflated 40%)
2023-01-11T22:56:22.3498565Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214645.xml (deflated 40%)
2023-01-11T22:56:22.3499330Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214652.xml (deflated 40%)
2023-01-11T22:56:22.3500106Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214700.xml (deflated 42%)
2023-01-11T22:56:22.3500881Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214703.xml (deflated 42%)
2023-01-11T22:56:22.3501708Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214705.xml (deflated 42%)
2023-01-11T22:56:22.3502514Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214707.xml (deflated 42%)
2023-01-11T22:56:22.3503301Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214710.xml (deflated 41%)
2023-01-11T22:56:22.3504078Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214734.xml (deflated 40%)
2023-01-11T22:56:22.3504867Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214758.xml (deflated 42%)
2023-01-11T22:56:22.3505626Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214800.xml (deflated 41%)
2023-01-11T22:56:22.3506406Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214803.xml (deflated 41%)
2023-01-11T22:56:22.3507182Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214805.xml (deflated 41%)
2023-01-11T22:56:22.3508017Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214807.xml (deflated 41%)
2023-01-11T22:56:22.3508775Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214810.xml (deflated 41%)
2023-01-11T22:56:22.3509552Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214818.xml (deflated 41%)
2023-01-11T22:56:22.3510321Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214825.xml (deflated 41%)
2023-01-11T22:56:22.3511095Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214833.xml (deflated 41%)
2023-01-11T22:56:22.3512113Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214841.xml (deflated 42%)
2023-01-11T22:56:22.3513031Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214845.xml (deflated 42%)
2023-01-11T22:56:22.3513814Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214848.xml (deflated 42%)
2023-01-11T22:56:22.3514594Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214850.xml (deflated 41%)
2023-01-11T22:56:22.3515356Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214853.xml (deflated 40%)
2023-01-11T22:56:22.3516143Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214859.xml (deflated 40%)
2023-01-11T22:56:22.3516921Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214906.xml (deflated 41%)
2023-01-11T22:56:22.3517701Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214909.xml (deflated 42%)
2023-01-11T22:56:22.3518463Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214911.xml (deflated 42%)
2023-01-11T22:56:22.3519241Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214915.xml (deflated 40%)
2023-01-11T22:56:22.3520018Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214924.xml (deflated 40%)
2023-01-11T22:56:22.3520791Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214932.xml (deflated 40%)
2023-01-11T22:56:22.3521550Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214941.xml (deflated 42%)
2023-01-11T22:56:22.3522423Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214945.xml (deflated 41%)
2023-01-11T22:56:22.3523220Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214949.xml (deflated 40%)
2023-01-11T22:56:22.3523997Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214956.xml (deflated 40%)
2023-01-11T22:56:22.3524753Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215003.xml (deflated 42%)
2023-01-11T22:56:22.3525768Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215007.xml (deflated 41%)
2023-01-11T22:56:22.3526679Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215009.xml (deflated 41%)
2023-01-11T22:56:22.3527671Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215012.xml (deflated 42%)
2023-01-11T22:56:22.3528436Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215014.xml (deflated 42%)
2023-01-11T22:56:22.3529214Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215016.xml (deflated 41%)
2023-01-11T22:56:22.3529987Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215019.xml (deflated 41%)
2023-01-11T22:56:22.3530758Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215021.xml (deflated 42%)
2023-01-11T22:56:22.3531535Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215024.xml (deflated 41%)
2023-01-11T22:56:22.3532301Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215026.xml (deflated 41%)
2023-01-11T22:56:22.3533081Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215028.xml (deflated 42%)
2023-01-11T22:56:22.3533858Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215031.xml (deflated 41%)
2023-01-11T22:56:22.3534631Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215039.xml (deflated 42%)
2023-01-11T22:56:22.3535395Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215041.xml (deflated 40%)
2023-01-11T22:56:22.3536166Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215050.xml (deflated 41%)
2023-01-11T22:56:22.3536938Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215059.xml (deflated 41%)
2023-01-11T22:56:22.3537721Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215101.xml (deflated 41%)
2023-01-11T22:56:22.3538483Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215109.xml (deflated 41%)
2023-01-11T22:56:22.3539257Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215118.xml (deflated 42%)
2023-01-11T22:56:22.3540030Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215120.xml (deflated 42%)
2023-01-11T22:56:22.3540803Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215122.xml (deflated 42%)
2023-01-11T22:56:22.3541566Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215125.xml (deflated 41%)
2023-01-11T22:56:22.3542409Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215127.xml (deflated 41%)
2023-01-11T22:56:22.3543191Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215136.xml (deflated 41%)
2023-01-11T22:56:22.3543969Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215145.xml (deflated 42%)
2023-01-11T22:56:22.3544935Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215148.xml (deflated 42%)
2023-01-11T22:56:22.3545818Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215150.xml (deflated 41%)
2023-01-11T22:56:22.3546705Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215153.xml (deflated 43%)
2023-01-11T22:56:22.3547568Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215155.xml (deflated 42%)
2023-01-11T22:56:22.3548330Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215157.xml (deflated 42%)
2023-01-11T22:56:22.3549107Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215200.xml (deflated 42%)
2023-01-11T22:56:22.3549881Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215202.xml (deflated 43%)
2023-01-11T22:56:22.3550655Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215205.xml (deflated 41%)
2023-01-11T22:56:22.3551411Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215212.xml (deflated 40%)
2023-01-11T22:56:22.3552560Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215221.xml (deflated 40%)
2023-01-11T22:56:22.3553346Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215229.xml (deflated 43%)
2023-01-11T22:56:22.3554126Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215231.xml (deflated 43%)
2023-01-11T22:56:22.3554886Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215234.xml (deflated 42%)
2023-01-11T22:56:22.3555665Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215236.xml (deflated 43%)
2023-01-11T22:56:22.3556434Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215238.xml (deflated 42%)
2023-01-11T22:56:22.3557216Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215241.xml (deflated 42%)
2023-01-11T22:56:22.3557977Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215243.xml (deflated 41%)
2023-01-11T22:56:22.3558761Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215252.xml (deflated 41%)
2023-01-11T22:56:22.3559533Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215254.xml (deflated 40%)
2023-01-11T22:56:22.3560310Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215302.xml (deflated 41%)
2023-01-11T22:56:22.3561067Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215311.xml (deflated 40%)
2023-01-11T22:56:22.3561932Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215325.xml (deflated 40%)
2023-01-11T22:56:22.3562723Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215341.xml (deflated 41%)
2023-01-11T22:56:22.3563499Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215349.xml (deflated 41%)
2023-01-11T22:56:22.3564263Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215351.xml (deflated 42%)
2023-01-11T22:56:22.3565042Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215358.xml (deflated 43%)
2023-01-11T22:56:22.3565824Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215402.xml (deflated 42%)
2023-01-11T22:56:22.3566598Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215405.xml (deflated 42%)
2023-01-11T22:56:22.3567433Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215407.xml (deflated 40%)
2023-01-11T22:56:22.3568207Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215416.xml (deflated 40%)
2023-01-11T22:56:22.3568978Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215425.xml (deflated 40%)
2023-01-11T22:56:22.3569757Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215433.xml (deflated 39%)
2023-01-11T22:56:22.3570515Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215441.xml (deflated 39%)
2023-01-11T22:56:22.3571296Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215450.xml (deflated 40%)
2023-01-11T22:56:22.3572083Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215459.xml (deflated 40%)
2023-01-11T22:56:22.3572857Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215507.xml (deflated 42%)
2023-01-11T22:56:22.3573616Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215511.xml (deflated 41%)
2023-01-11T22:56:22.3574390Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215518.xml (deflated 42%)
2023-01-11T22:56:22.3575164Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215523.xml (deflated 42%)
2023-01-11T22:56:22.3575936Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215531.xml (deflated 42%)
2023-01-11T22:56:22.3576720Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215533.xml (deflated 46%)
2023-01-11T22:56:22.3577475Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215542.xml (deflated 47%)
2023-01-11T22:56:22.3578244Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215550.xml (deflated 48%)
2023-01-11T22:56:22.3579018Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215559.xml (deflated 45%)
2023-01-11T22:56:22.3579787Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215607.xml (deflated 41%)
2023-01-11T22:56:22.3580552Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215610.xml (deflated 44%)
2023-01-11T22:56:22.3581382Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215612.xml (deflated 43%)
2023-01-11T22:56:22.3582168Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215615.xml (deflated 43%)
2023-01-11T22:56:22.3582944Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215617.xml (deflated 43%)
2023-01-11T22:56:22.3583705Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215619.xml (deflated 43%)
2023-01-11T22:56:22.3584480Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215622.xml (deflated 41%)
2023-01-11T22:56:22.3585255Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215624.xml (deflated 40%)
2023-01-11T22:56:22.3586085Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215633.xml (deflated 40%)
2023-01-11T22:56:22.3586844Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215642.xml (deflated 41%)
2023-01-11T22:56:22.3587618Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215644.xml (deflated 41%)
2023-01-11T22:56:22.3588399Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215647.xml (deflated 40%)
2023-01-11T22:56:22.3589173Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215656.xml (deflated 40%)
2023-01-11T22:56:22.3589934Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215705.xml (deflated 42%)
2023-01-11T22:56:22.3590710Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215709.xml (deflated 42%)
2023-01-11T22:56:22.3591496Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215714.xml (deflated 40%)
2023-01-11T22:56:22.3592714Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215722.xml (deflated 40%)
2023-01-11T22:56:22.3593487Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215731.xml (deflated 40%)
2023-01-11T22:56:22.3594265Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215739.xml (deflated 41%)
2023-01-11T22:56:22.3595041Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215742.xml (deflated 41%)
2023-01-11T22:56:22.3595818Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215744.xml (deflated 41%)
2023-01-11T22:56:22.3596592Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215746.xml (deflated 41%)
2023-01-11T22:56:22.3597369Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215749.xml (deflated 40%)
2023-01-11T22:56:22.3598146Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215751.xml (deflated 41%)
2023-01-11T22:56:22.3598922Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215753.xml (deflated 41%)
2023-01-11T22:56:22.3599682Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215756.xml (deflated 41%)
2023-01-11T22:56:22.3600463Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215758.xml (deflated 41%)
2023-01-11T22:56:22.3601373Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215801.xml (deflated 41%)
2023-01-11T22:56:22.3602168Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215803.xml (deflated 41%)
2023-01-11T22:56:22.3602926Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215805.xml (deflated 41%)
2023-01-11T22:56:22.3603698Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215808.xml (deflated 41%)
2023-01-11T22:56:22.3604469Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215810.xml (deflated 42%)
2023-01-11T22:56:22.3605246Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215812.xml (deflated 41%)
2023-01-11T22:56:22.3606076Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215815.xml (deflated 41%)
2023-01-11T22:56:22.3606858Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215817.xml (deflated 41%)
2023-01-11T22:56:22.3607636Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215820.xml (deflated 41%)
2023-01-11T22:56:22.3608414Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215822.xml (deflated 41%)
2023-01-11T22:56:22.3609164Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215824.xml (deflated 41%)
2023-01-11T22:56:22.3609937Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215827.xml (deflated 41%)
2023-01-11T22:56:22.3610719Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215829.xml (deflated 42%)
2023-01-11T22:56:22.3611584Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215832.xml (deflated 42%)
2023-01-11T22:56:22.3612851Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215834.xml (deflated 41%)
2023-01-11T22:56:22.3614329Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215836.xml (deflated 43%)
2023-01-11T22:56:22.3615151Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215839.xml (deflated 42%)
2023-01-11T22:56:22.3615918Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215841.xml (deflated 41%)
2023-01-11T22:56:22.3616720Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215844.xml (deflated 41%)
2023-01-11T22:56:22.3617501Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215852.xml (deflated 41%)
2023-01-11T22:56:22.3618275Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215854.xml (deflated 41%)
2023-01-11T22:56:22.3619035Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215857.xml (deflated 41%)
2023-01-11T22:56:22.3619812Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215859.xml (deflated 41%)
2023-01-11T22:56:22.3620581Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215907.xml (deflated 40%)
2023-01-11T22:56:22.3621360Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215916.xml (deflated 41%)
2023-01-11T22:56:22.3622217Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215924.xml (deflated 42%)
2023-01-11T22:56:22.3623008Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215926.xml (deflated 42%)
2023-01-11T22:56:22.3623778Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215929.xml (deflated 41%)
2023-01-11T22:56:22.3624552Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215937.xml (deflated 40%)
2023-01-11T22:56:22.3625314Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215945.xml (deflated 42%)
2023-01-11T22:56:22.3626092Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215947.xml (deflated 41%)
2023-01-11T22:56:22.3626928Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215955.xml (deflated 42%)
2023-01-11T22:56:22.3627703Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215958.xml (deflated 42%)
2023-01-11T22:56:22.3628464Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220005.xml (deflated 43%)
2023-01-11T22:56:22.3629243Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220007.xml (deflated 42%)
2023-01-11T22:56:22.3630018Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220009.xml (deflated 43%)
2023-01-11T22:56:22.3630793Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220014.xml (deflated 41%)
2023-01-11T22:56:22.3631559Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220023.xml (deflated 43%)
2023-01-11T22:56:22.3632735Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220025.xml (deflated 42%)
2023-01-11T22:56:22.3633523Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220029.xml (deflated 42%)
2023-01-11T22:56:22.3634300Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220032.xml (deflated 42%)
2023-01-11T22:56:22.3635058Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220038.xml (deflated 42%)
2023-01-11T22:56:22.3635841Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220041.xml (deflated 42%)
2023-01-11T22:56:22.3636627Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220043.xml (deflated 41%)
2023-01-11T22:56:22.3637406Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220051.xml (deflated 41%)
2023-01-11T22:56:22.3638167Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220059.xml (deflated 42%)
2023-01-11T22:56:22.3638946Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220102.xml (deflated 41%)
2023-01-11T22:56:22.3639716Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220110.xml (deflated 43%)
2023-01-11T22:56:22.3640486Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220112.xml (deflated 42%)
2023-01-11T22:56:22.3641349Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220119.xml (deflated 41%)
2023-01-11T22:56:22.3642142Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220128.xml (deflated 42%)
2023-01-11T22:56:22.3642918Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220134.xml (deflated 42%)
2023-01-11T22:56:22.3643695Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220137.xml (deflated 42%)
2023-01-11T22:56:22.3644451Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220139.xml (deflated 41%)
2023-01-11T22:56:22.3645226Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220148.xml (deflated 42%)
2023-01-11T22:56:22.3646080Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220150.xml (deflated 42%)
2023-01-11T22:56:22.3646854Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220157.xml (deflated 42%)
2023-01-11T22:56:22.3647615Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220200.xml (deflated 42%)
2023-01-11T22:56:22.3648394Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220206.xml (deflated 42%)
2023-01-11T22:56:22.3649310Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220209.xml (deflated 43%)
2023-01-11T22:56:22.3650250Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220211.xml (deflated 43%)
2023-01-11T22:56:22.3651028Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220213.xml (deflated 41%)
2023-01-11T22:56:22.3651803Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220216.xml (deflated 41%)
2023-01-11T22:56:22.3652577Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220218.xml (deflated 41%)
2023-01-11T22:56:22.3653352Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220225.xml (deflated 41%)
2023-01-11T22:56:22.3654127Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220233.xml (deflated 42%)
2023-01-11T22:56:22.3654891Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220240.xml (deflated 41%)
2023-01-11T22:56:22.3655669Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220247.xml (deflated 41%)
2023-01-11T22:56:22.3656448Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220254.xml (deflated 41%)
2023-01-11T22:56:22.3657223Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220302.xml (deflated 41%)
2023-01-11T22:56:22.3657985Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220310.xml (deflated 42%)
2023-01-11T22:56:22.3658761Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220317.xml (deflated 42%)
2023-01-11T22:56:22.3659531Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220320.xml (deflated 42%)
2023-01-11T22:56:22.3660307Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220324.xml (deflated 42%)
2023-01-11T22:56:22.3661152Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220326.xml (deflated 42%)
2023-01-11T22:56:22.3661938Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220329.xml (deflated 44%)
2023-01-11T22:56:22.3662714Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220331.xml (deflated 42%)
2023-01-11T22:56:22.3663490Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220335.xml (deflated 42%)
2023-01-11T22:56:22.3664251Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220339.xml (deflated 42%)
2023-01-11T22:56:22.3665026Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220344.xml (deflated 41%)
2023-01-11T22:56:22.3665855Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220353.xml (deflated 40%)
2023-01-11T22:56:22.3666638Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220402.xml (deflated 40%)
2023-01-11T22:56:22.3667394Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220411.xml (deflated 40%)
2023-01-11T22:56:22.3668175Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220419.xml (deflated 40%)
2023-01-11T22:56:22.3668948Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220428.xml (deflated 42%)
2023-01-11T22:56:22.3669721Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220432.xml (deflated 42%)
2023-01-11T22:56:22.3670485Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220436.xml (deflated 40%)
2023-01-11T22:56:22.3671259Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220444.xml (deflated 40%)
2023-01-11T22:56:22.3672377Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220452.xml (deflated 40%)
2023-01-11T22:56:22.3673182Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220501.xml (deflated 40%)
2023-01-11T22:56:22.3673943Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220509.xml (deflated 42%)
2023-01-11T22:56:22.3674721Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220513.xml (deflated 43%)
2023-01-11T22:56:22.3675503Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220516.xml (deflated 42%)
2023-01-11T22:56:22.3676275Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220520.xml (deflated 41%)
2023-01-11T22:56:22.3677035Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220529.xml (deflated 42%)
2023-01-11T22:56:22.3677811Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220533.xml (deflated 42%)
2023-01-11T22:56:22.3678587Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220537.xml (deflated 40%)
2023-01-11T22:56:22.3679368Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220546.xml (deflated 40%)
2023-01-11T22:56:22.3680124Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220555.xml (deflated 42%)
2023-01-11T22:56:22.3680989Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220559.xml (deflated 40%)
2023-01-11T22:56:22.3681777Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220608.xml (deflated 42%)
2023-01-11T22:56:22.3682547Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220611.xml (deflated 42%)
2023-01-11T22:56:22.3683301Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220613.xml (deflated 42%)
2023-01-11T22:56:22.3684080Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220615.xml (deflated 41%)
2023-01-11T22:56:22.3684852Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220618.xml (deflated 41%)
2023-01-11T22:56:22.3685699Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220620.xml (deflated 41%)
2023-01-11T22:56:22.3686458Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220623.xml (deflated 41%)
2023-01-11T22:56:22.3687240Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220625.xml (deflated 41%)
2023-01-11T22:56:22.3688010Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220627.xml (deflated 41%)
2023-01-11T22:56:22.3688782Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220630.xml (deflated 41%)
2023-01-11T22:56:22.3689541Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220632.xml (deflated 42%)
2023-01-11T22:56:22.3690323Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220634.xml (deflated 42%)
2023-01-11T22:56:22.3691095Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220637.xml (deflated 42%)
2023-01-11T22:56:22.3691865Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220641.xml (deflated 40%)
2023-01-11T22:56:22.3692625Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220650.xml (deflated 41%)
2023-01-11T22:56:22.3693399Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220658.xml (deflated 41%)
2023-01-11T22:56:22.3694171Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220701.xml (deflated 40%)
2023-01-11T22:56:22.3694951Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220709.xml (deflated 40%)
2023-01-11T22:56:22.3695711Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220717.xml (deflated 40%)
2023-01-11T22:56:22.3696492Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220735.xml (deflated 41%)
2023-01-11T22:56:22.3697260Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220744.xml (deflated 41%)
2023-01-11T22:56:22.3698029Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220753.xml (deflated 41%)
2023-01-11T22:56:22.3698802Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220801.xml (deflated 41%)
2023-01-11T22:56:22.3699630Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220810.xml (deflated 42%)
2023-01-11T22:56:22.3700408Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220814.xml (deflated 42%)
2023-01-11T22:56:22.3701234Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220818.xml (deflated 42%)
2023-01-11T22:56:22.3702012Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220827.xml (deflated 41%)
2023-01-11T22:56:22.3702772Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220835.xml (deflated 42%)
2023-01-11T22:56:22.3703548Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220839.xml (deflated 40%)
2023-01-11T22:56:22.3704377Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220848.xml (deflated 42%)
2023-01-11T22:56:22.3705158Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220852.xml (deflated 41%)
2023-01-11T22:56:22.3705923Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220901.xml (deflated 41%)
2023-01-11T22:56:22.3706697Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220909.xml (deflated 40%)
2023-01-11T22:56:22.3707469Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220917.xml (deflated 42%)
2023-01-11T22:56:22.3708246Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220921.xml (deflated 42%)
2023-01-11T22:56:22.3709007Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220925.xml (deflated 42%)
2023-01-11T22:56:22.3709786Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220930.xml (deflated 41%)
2023-01-11T22:56:22.3710556Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220938.xml (deflated 41%)
2023-01-11T22:56:22.3711334Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220946.xml (deflated 40%)
2023-01-11T22:56:22.3712435Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220953.xml (deflated 41%)
2023-01-11T22:56:22.3713238Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221000.xml (deflated 42%)
2023-01-11T22:56:22.3714009Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221005.xml (deflated 42%)
2023-01-11T22:56:22.3714795Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221009.xml (deflated 41%)
2023-01-11T22:56:22.3715551Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221016.xml (deflated 42%)
2023-01-11T22:56:22.3716327Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221018.xml (deflated 42%)
2023-01-11T22:56:22.3717095Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221020.xml (deflated 41%)
2023-01-11T22:56:22.3717866Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221029.xml (deflated 42%)
2023-01-11T22:56:22.3718618Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221032.xml (deflated 42%)
2023-01-11T22:56:22.3719470Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221034.xml (deflated 40%)
2023-01-11T22:56:22.3720259Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221042.xml (deflated 42%)
2023-01-11T22:56:22.3721033Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221046.xml (deflated 41%)
2023-01-11T22:56:22.3721788Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221053.xml (deflated 41%)
2023-01-11T22:56:22.3722561Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221102.xml (deflated 41%)
2023-01-11T22:56:22.3723331Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221108.xml (deflated 40%)
2023-01-11T22:56:22.3724174Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221115.xml (deflated 40%)
2023-01-11T22:56:22.3724931Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221122.xml (deflated 40%)
2023-01-11T22:56:22.3725704Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221130.xml (deflated 43%)
2023-01-11T22:56:22.3726477Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221133.xml (deflated 43%)
2023-01-11T22:56:22.3727249Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221135.xml (deflated 42%)
2023-01-11T22:56:22.3728005Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221138.xml (deflated 42%)
2023-01-11T22:56:22.3728793Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221140.xml (deflated 41%)
2023-01-11T22:56:22.3729568Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221204.xml (deflated 41%)
2023-01-11T22:56:22.3730342Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221228.xml (deflated 42%)
2023-01-11T22:56:22.3731101Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221231.xml (deflated 42%)
2023-01-11T22:56:22.3731878Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221233.xml (deflated 41%)
2023-01-11T22:56:22.3732650Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221236.xml (deflated 41%)
2023-01-11T22:56:22.3733427Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221238.xml (deflated 41%)
2023-01-11T22:56:22.3734189Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221240.xml (deflated 41%)
2023-01-11T22:56:22.3734966Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221248.xml (deflated 41%)
2023-01-11T22:56:22.3735744Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221256.xml (deflated 41%)
2023-01-11T22:56:22.3736518Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221304.xml (deflated 41%)
2023-01-11T22:56:22.3737281Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221312.xml (deflated 42%)
2023-01-11T22:56:22.3738057Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221316.xml (deflated 42%)
2023-01-11T22:56:22.3738884Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221318.xml (deflated 42%)
2023-01-11T22:56:22.3739666Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221321.xml (deflated 42%)
2023-01-11T22:56:22.3740420Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221323.xml (deflated 40%)
2023-01-11T22:56:22.3741197Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221330.xml (deflated 41%)
2023-01-11T22:56:22.3741972Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221337.xml (deflated 41%)
2023-01-11T22:56:22.3742745Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221339.xml (deflated 42%)
2023-01-11T22:56:22.3743554Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221342.xml (deflated 42%)
2023-01-11T22:56:22.3744336Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221346.xml (deflated 40%)
2023-01-11T22:56:22.3745170Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221355.xml (deflated 41%)
2023-01-11T22:56:22.3745945Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221403.xml (deflated 40%)
2023-01-11T22:56:22.3746719Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221411.xml (deflated 42%)
2023-01-11T22:56:22.3747479Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221415.xml (deflated 41%)
2023-01-11T22:56:22.3748255Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221419.xml (deflated 40%)
2023-01-11T22:56:22.3749031Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221426.xml (deflated 40%)
2023-01-11T22:56:22.3749809Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221433.xml (deflated 42%)
2023-01-11T22:56:22.3750569Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221437.xml (deflated 41%)
2023-01-11T22:56:22.3751344Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221440.xml (deflated 42%)
2023-01-11T22:56:22.3752469Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221442.xml (deflated 42%)
2023-01-11T22:56:22.3753279Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221444.xml (deflated 42%)
2023-01-11T22:56:22.3754041Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221447.xml (deflated 41%)
2023-01-11T22:56:22.3754821Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221449.xml (deflated 41%)
2023-01-11T22:56:22.3755593Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221452.xml (deflated 42%)
2023-01-11T22:56:22.3756369Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221454.xml (deflated 41%)
2023-01-11T22:56:22.3757123Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221456.xml (deflated 41%)
2023-01-11T22:56:22.3757980Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221459.xml (deflated 41%)
2023-01-11T22:56:22.3758771Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221501.xml (deflated 41%)
2023-01-11T22:56:22.3759550Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221509.xml (deflated 42%)
2023-01-11T22:56:22.3760309Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221512.xml (deflated 41%)
2023-01-11T22:56:22.3761083Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221521.xml (deflated 41%)
2023-01-11T22:56:22.3761852Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221529.xml (deflated 41%)
2023-01-11T22:56:22.3762632Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221531.xml (deflated 41%)
2023-01-11T22:56:22.3763458Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221540.xml (deflated 41%)
2023-01-11T22:56:22.3764233Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221548.xml (deflated 41%)
2023-01-11T22:56:22.3765004Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221550.xml (deflated 42%)
2023-01-11T22:56:22.3765782Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221553.xml (deflated 41%)
2023-01-11T22:56:22.3766541Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221555.xml (deflated 42%)
2023-01-11T22:56:22.3767315Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221557.xml (deflated 40%)
2023-01-11T22:56:22.3768096Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221606.xml (deflated 41%)
2023-01-11T22:56:22.3768870Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221615.xml (deflated 42%)
2023-01-11T22:56:22.3769629Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221618.xml (deflated 42%)
2023-01-11T22:56:22.3770399Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221620.xml (deflated 41%)
2023-01-11T22:56:22.3771169Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221623.xml (deflated 43%)
2023-01-11T22:56:22.3771940Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221625.xml (deflated 42%)
2023-01-11T22:56:22.3772707Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221627.xml (deflated 42%)
2023-01-11T22:56:22.3773486Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221630.xml (deflated 42%)
2023-01-11T22:56:22.3774261Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221632.xml (deflated 43%)
2023-01-11T22:56:22.3775031Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221635.xml (deflated 41%)
2023-01-11T22:56:22.3775791Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221642.xml (deflated 40%)
2023-01-11T22:56:22.3776569Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221651.xml (deflated 41%)
2023-01-11T22:56:22.3777396Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221659.xml (deflated 43%)
2023-01-11T22:56:22.3778183Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221701.xml (deflated 43%)
2023-01-11T22:56:22.3778942Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221704.xml (deflated 43%)
2023-01-11T22:56:22.3779720Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221706.xml (deflated 43%)
2023-01-11T22:56:22.3780493Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221709.xml (deflated 42%)
2023-01-11T22:56:22.3781267Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221711.xml (deflated 42%)
2023-01-11T22:56:22.3782077Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221713.xml (deflated 42%)
2023-01-11T22:56:22.3782856Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221722.xml (deflated 41%)
2023-01-11T22:56:22.3783627Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221724.xml (deflated 40%)
2023-01-11T22:56:22.3784399Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221733.xml (deflated 41%)
2023-01-11T22:56:22.3785152Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221741.xml (deflated 41%)
2023-01-11T22:56:22.3785931Z   adding: test/test-reports/dist-nccl/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221755.xml (deflated 40%)
2023-01-11T22:56:22.3806358Z ##[group]Run # Remove any previous test reports if they exist
2023-01-11T22:56:22.3806734Z # Remove any previous test reports if they exist
2023-01-11T22:56:22.3807048Z rm -f usage-log-*.zip
2023-01-11T22:56:22.3807414Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2023-01-11T22:56:22.3807803Z # so check to see if the file exists first
2023-01-11T22:56:22.3808087Z if [ -f 'usage_log.txt' ]; then
2023-01-11T22:56:22.3808414Z     zip "usage-log-${FILE_SUFFIX}.zip" 'usage_log.txt'
2023-01-11T22:56:22.3808704Z fi
2023-01-11T22:56:22.3820460Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T22:56:22.3820750Z env:
2023-01-11T22:56:22.3820986Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:56:22.3821229Z   GPU_FLAG: --gpus all
2023-01-11T22:56:22.3821602Z   DOCKER_CONTAINER_ID: c5cb3f34fdb745cbab81ead9bbd8979c6cd118fca3d50c4cb582659c15cdaef3
2023-01-11T22:56:22.3822098Z   FILE_SUFFIX: test-distributed-2-3-linux.8xlarge.nvidia.gpu_10589560479
2023-01-11T22:56:22.3822435Z ##[endgroup]
2023-01-11T22:56:22.4647830Z   adding: usage_log.txt (deflated 95%)
2023-01-11T22:56:22.4721103Z ##[group]Run seemethere/upload-artifact-s3@v5
2023-01-11T22:56:22.4721371Z with:
2023-01-11T22:56:22.4721640Z   s3-prefix: pytorch/pytorch/3896346758/1/artifact

2023-01-11T22:56:22.4721929Z   retention-days: 14
2023-01-11T22:56:22.4722170Z   if-no-files-found: warn
2023-01-11T22:56:22.4722429Z   path: test-jsons-*.zip
2023-01-11T22:56:22.4722671Z   name: artifact
2023-01-11T22:56:22.4722890Z   s3-bucket: gha-artifacts
2023-01-11T22:56:22.4723143Z   region: us-east-1
2023-01-11T22:56:22.4723366Z env:
2023-01-11T22:56:22.4723580Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:56:22.4723833Z   GPU_FLAG: --gpus all
2023-01-11T22:56:22.4724203Z   DOCKER_CONTAINER_ID: c5cb3f34fdb745cbab81ead9bbd8979c6cd118fca3d50c4cb582659c15cdaef3
2023-01-11T22:56:22.4724551Z ##[endgroup]
2023-01-11T22:56:22.9158481Z NOTE: s3-prefix specified, ignoring name parameter
2023-01-11T22:56:22.9159543Z With the provided path, there will be 1 file uploaded
2023-01-11T22:56:22.9159930Z Uploading to s3 prefix: pytorch/pytorch/3896346758/1/artifact
2023-01-11T22:56:22.9170881Z Starting upload of test-jsons-test-distributed-2-3-linux.8xlarge.nvidia.gpu_10589560479.zip
2023-01-11T22:56:23.0781332Z Finished upload of test-jsons-test-distributed-2-3-linux.8xlarge.nvidia.gpu_10589560479.zip
2023-01-11T22:56:23.0943619Z ##[group]Run seemethere/upload-artifact-s3@v5
2023-01-11T22:56:23.0944037Z with:
2023-01-11T22:56:23.0944479Z   s3-prefix: pytorch/pytorch/3896346758/1/artifact

2023-01-11T22:56:23.0944763Z   retention-days: 14
2023-01-11T22:56:23.0945032Z   if-no-files-found: error
2023-01-11T22:56:23.0945313Z   path: test-reports-*.zip
2023-01-11T22:56:23.0945553Z   name: artifact
2023-01-11T22:56:23.0945805Z   s3-bucket: gha-artifacts
2023-01-11T22:56:23.0946065Z   region: us-east-1
2023-01-11T22:56:23.0946274Z env:
2023-01-11T22:56:23.0946668Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:56:23.0946934Z   GPU_FLAG: --gpus all
2023-01-11T22:56:23.0947307Z   DOCKER_CONTAINER_ID: c5cb3f34fdb745cbab81ead9bbd8979c6cd118fca3d50c4cb582659c15cdaef3
2023-01-11T22:56:23.0947675Z ##[endgroup]
2023-01-11T22:56:23.5339136Z NOTE: s3-prefix specified, ignoring name parameter
2023-01-11T22:56:23.5340251Z With the provided path, there will be 1 file uploaded
2023-01-11T22:56:23.5340610Z Uploading to s3 prefix: pytorch/pytorch/3896346758/1/artifact
2023-01-11T22:56:23.5352507Z Starting upload of test-reports-test-distributed-2-3-linux.8xlarge.nvidia.gpu_10589560479.zip
2023-01-11T22:56:23.8178480Z Finished upload of test-reports-test-distributed-2-3-linux.8xlarge.nvidia.gpu_10589560479.zip
2023-01-11T22:56:23.8331728Z ##[group]Run seemethere/upload-artifact-s3@v5
2023-01-11T22:56:23.8332018Z with:
2023-01-11T22:56:23.8332290Z   s3-prefix: pytorch/pytorch/3896346758/1/artifact

2023-01-11T22:56:23.8332567Z   retention-days: 14
2023-01-11T22:56:23.8332843Z   if-no-files-found: ignore
2023-01-11T22:56:23.8333114Z   path: usage-log-*.zip
2023-01-11T22:56:23.8333350Z   name: artifact
2023-01-11T22:56:23.8333596Z   s3-bucket: gha-artifacts
2023-01-11T22:56:23.8333852Z   region: us-east-1
2023-01-11T22:56:23.8334060Z env:
2023-01-11T22:56:23.8334290Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:56:23.8334552Z   GPU_FLAG: --gpus all
2023-01-11T22:56:23.8334914Z   DOCKER_CONTAINER_ID: c5cb3f34fdb745cbab81ead9bbd8979c6cd118fca3d50c4cb582659c15cdaef3
2023-01-11T22:56:23.8335277Z ##[endgroup]
2023-01-11T22:56:24.2814635Z NOTE: s3-prefix specified, ignoring name parameter
2023-01-11T22:56:24.2815367Z With the provided path, there will be 1 file uploaded
2023-01-11T22:56:24.2815739Z Uploading to s3 prefix: pytorch/pytorch/3896346758/1/artifact
2023-01-11T22:56:24.2827139Z Starting upload of usage-log-test-distributed-2-3-linux.8xlarge.nvidia.gpu_10589560479.zip
2023-01-11T22:56:24.4720099Z Finished upload of usage-log-test-distributed-2-3-linux.8xlarge.nvidia.gpu_10589560479.zip
2023-01-11T22:56:24.4870455Z ##[group]Run # shellcheck disable=SC2156
2023-01-11T22:56:24.4870820Z # shellcheck disable=SC2156
2023-01-11T22:56:24.4871217Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2023-01-11T22:56:24.4885165Z shell: /usr/bin/bash -e {0}
2023-01-11T22:56:24.4885421Z env:
2023-01-11T22:56:24.4885660Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:56:24.4885908Z   GPU_FLAG: --gpus all
2023-01-11T22:56:24.4886291Z   DOCKER_CONTAINER_ID: c5cb3f34fdb745cbab81ead9bbd8979c6cd118fca3d50c4cb582659c15cdaef3
2023-01-11T22:56:24.4886661Z ##[endgroup]
2023-01-11T22:56:24.8153346Z ##[group]Run set -x
2023-01-11T22:56:24.8153757Z set -x
2023-01-11T22:56:24.8154051Z python3 -m pip install -r requirements.txt
2023-01-11T22:56:24.8154393Z python3 -m pip install boto3==1.19.12
2023-01-11T22:56:24.8154789Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2023-01-11T22:56:24.8168030Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T22:56:24.8168327Z env:
2023-01-11T22:56:24.8168567Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:56:24.8168814Z   GPU_FLAG: --gpus all
2023-01-11T22:56:24.8169195Z   DOCKER_CONTAINER_ID: c5cb3f34fdb745cbab81ead9bbd8979c6cd118fca3d50c4cb582659c15cdaef3
2023-01-11T22:56:24.8169581Z   AWS_DEFAULT_REGION: us-east-1
2023-01-11T22:56:24.8169813Z   BRANCH: 
2023-01-11T22:56:24.8170048Z   TEST_CONFIG: distributed
2023-01-11T22:56:24.8170298Z   SHARD_NUMBER: 2
2023-01-11T22:56:24.8170594Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.7-py3.10-gcc7
2023-01-11T22:56:24.8170942Z   PR_NUMBER: 
2023-01-11T22:56:24.8171200Z   PYTORCH_RETRY_TEST_CASES: 1
2023-01-11T22:56:24.8171465Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2023-01-11T22:56:24.8171780Z   SHA1: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T22:56:24.8172071Z   TAG: ciflow/trunk/91627
2023-01-11T22:56:24.8172307Z   WORKFLOW_ID: 3896346758
2023-01-11T22:56:24.8172879Z   GITHUB_TOKEN: ***
2023-01-11T22:56:24.8173139Z   GHA_WORKFLOW_JOB_ID: 10589560479
2023-01-11T22:56:24.8173376Z ##[endgroup]
2023-01-11T22:56:24.8202391Z + python3 -m pip install -r requirements.txt
2023-01-11T22:56:25.1114285Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T22:56:25.1466507Z Requirement already satisfied: astunparse in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (1.6.3)
2023-01-11T22:56:25.1504845Z Requirement already satisfied: expecttest in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (0.1.4)
2023-01-11T22:56:25.1516447Z Requirement already satisfied: future in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (0.18.2)
2023-01-11T22:56:25.1528209Z Requirement already satisfied: hypothesis in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 5)) (6.62.0)
2023-01-11T22:56:25.2056188Z Requirement already satisfied: numpy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 6)) (1.21.6)
2023-01-11T22:56:25.2068312Z Requirement already satisfied: psutil in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (5.9.1)
2023-01-11T22:56:25.2176606Z Requirement already satisfied: pyyaml in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 8)) (6.0)
2023-01-11T22:56:25.2187570Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (2.26.0)
2023-01-11T22:56:25.2431782Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (49.1.3)
2023-01-11T22:56:25.2671705Z Requirement already satisfied: six in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 11)) (1.16.0)
2023-01-11T22:56:25.2683729Z Requirement already satisfied: types-dataclasses in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 12)) (0.6.6)
2023-01-11T22:56:25.2691766Z Requirement already satisfied: typing_extensions in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 13)) (4.4.0)
2023-01-11T22:56:25.2705355Z Requirement already satisfied: sympy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 14)) (1.10.1)
2023-01-11T22:56:25.2731257Z Requirement already satisfied: filelock in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 15)) (3.9.0)
2023-01-11T22:56:25.2833153Z Requirement already satisfied: networkx in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 16)) (2.6.3)
2023-01-11T22:56:25.3058737Z Requirement already satisfied: jinja2 in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 17)) (3.1.2)
2023-01-11T22:56:25.3093652Z Requirement already satisfied: wheel<1.0,>=0.23.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from astunparse->-r requirements.txt (line 2)) (0.38.4)
2023-01-11T22:56:25.3116134Z Requirement already satisfied: sortedcontainers<3.0.0,>=2.1.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (2.4.0)
2023-01-11T22:56:25.3129852Z Requirement already satisfied: attrs>=19.2.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (22.2.0)
2023-01-11T22:56:25.3496933Z Requirement already satisfied: exceptiongroup>=1.0.0; python_version < "3.11" in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (1.1.0)
2023-01-11T22:56:25.3520590Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2.0.12)
2023-01-11T22:56:25.3548875Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2022.12.7)
2023-01-11T22:56:25.3561340Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (3.4)
2023-01-11T22:56:25.3576999Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (1.26.14)
2023-01-11T22:56:25.3794488Z Requirement already satisfied: mpmath>=0.19 in /home/ec2-user/.local/lib/python3.7/site-packages (from sympy->-r requirements.txt (line 14)) (1.2.1)
2023-01-11T22:56:25.3870651Z Requirement already satisfied: MarkupSafe>=2.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from jinja2->-r requirements.txt (line 17)) (2.1.1)
2023-01-11T22:56:25.4538958Z + python3 -m pip install boto3==1.19.12
2023-01-11T22:56:25.7428928Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T22:56:25.7660088Z Requirement already satisfied: boto3==1.19.12 in /home/ec2-user/.local/lib/python3.7/site-packages (1.19.12)
2023-01-11T22:56:25.7731033Z Requirement already satisfied: botocore<1.23.0,>=1.22.12 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (1.22.12)
2023-01-11T22:56:25.7790680Z Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.10.0)
2023-01-11T22:56:25.7818471Z Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.5.2)
2023-01-11T22:56:25.7854471Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.14)
2023-01-11T22:56:25.8076475Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (2.8.2)
2023-01-11T22:56:25.8102035Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0)
2023-01-11T22:56:26.0586070Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2023-01-11T22:59:40.9538675Z [scribe] Scribe access token not provided, sending report via boto3...
2023-01-11T22:59:40.9542449Z ERROR ENCOUNTERED WHEN UPLOADING TO SCRIBE: {"errorMessage":"2023-01-11T22:59:26.997Z 72907258-5bb6-4b0e-b7c5-19c60a126fd5 Task timed out after 60.02 seconds"}
2023-01-11T22:59:40.9542818Z 
2023-01-11T22:59:40.9543390Z ----- Historic stats comparison result ------
2023-01-11T22:59:40.9543622Z 
2023-01-11T22:59:40.9543857Z     job: linux-bionic-cuda11.7-py3.10-gcc7
2023-01-11T22:59:40.9544213Z     commit: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T22:59:40.9552975Z 
2023-01-11T22:59:40.9553301Z Commit graph (base is most recent master ancestor with at least one S3 report):
2023-01-11T22:59:40.9553761Z 
2023-01-11T22:59:40.9554012Z     : (master)
2023-01-11T22:59:40.9554483Z     |
2023-01-11T22:59:40.9554988Z     | * 8419ddda87 (HEAD)            total time  2846.72s
2023-01-11T22:59:40.9555243Z     | |
2023-01-11T22:59:40.9555466Z     | : (2 commits)
2023-01-11T22:59:40.9555692Z     |/
2023-01-11T22:59:40.9560927Z     * db2a237763 (base)  11 reports, total time  4966.48s ± 3495.23s
2023-01-11T22:59:40.9561456Z     * 2b0abd4ce3         11 reports, total time  4990.75s ± 3463.36s
2023-01-11T22:59:40.9561891Z     * f7939b21e1         33 reports, total time  3500.97s ± 3537.57s
2023-01-11T22:59:40.9562373Z     * cb3204823e         11 reports, total time  4951.99s ± 3458.09s
2023-01-11T22:59:40.9562783Z     * 6e236553f5         11 reports, total time  4964.39s ± 3513.19s
2023-01-11T22:59:40.9563201Z     * cce577b391         11 reports, total time  4938.35s ± 3358.29s
2023-01-11T22:59:40.9563618Z     * fae821c2f1         11 reports, total time  4751.06s ± 3169.39s
2023-01-11T22:59:40.9564299Z     * 0c3659586d         11 reports, total time  4713.33s ± 3185.77s
2023-01-11T22:59:40.9564691Z     * 122245985a         11 reports, total time  4767.91s ± 3184.40s
2023-01-11T22:59:40.9565099Z     * b797a24259         11 reports, total time  4784.26s ± 3253.10s
2023-01-11T22:59:40.9565372Z     |
2023-01-11T22:59:40.9565560Z     :
2023-01-11T22:59:40.9565696Z 
2023-01-11T22:59:40.9565867Z Removed  (across 1387 suites)     0 tests, totaling     0.00s
2023-01-11T22:59:40.9566219Z Modified (across    0 suites)     0 tests, totaling     0.00s
2023-01-11T22:59:40.9566547Z Added    (across   78 suites)   983 tests, totaling +3616.78s
2023-01-11T22:59:41.0127096Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main
2023-01-11T22:59:41.0127468Z with:
2023-01-11T22:59:41.0127664Z env:
2023-01-11T22:59:41.0127901Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:59:41.0128168Z   GPU_FLAG: --gpus all
2023-01-11T22:59:41.0128535Z   DOCKER_CONTAINER_ID: c5cb3f34fdb745cbab81ead9bbd8979c6cd118fca3d50c4cb582659c15cdaef3
2023-01-11T22:59:41.0128916Z ##[endgroup]
2023-01-11T22:59:41.0146184Z ##[group]Run set -eou pipefail
2023-01-11T22:59:41.0146473Z set -eou pipefail
2023-01-11T22:59:41.0146720Z 
2023-01-11T22:59:41.0147042Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2023-01-11T22:59:41.0147362Z for _ in $(seq 1440); do
2023-01-11T22:59:41.0147658Z     # Break if no ssh session exists anymore
2023-01-11T22:59:41.0147955Z     if [ "$(who)" = "" ]; then
2023-01-11T22:59:41.0148211Z       break
2023-01-11T22:59:41.0148463Z     fi
2023-01-11T22:59:41.0148694Z     echo "."
2023-01-11T22:59:41.0148910Z     sleep 5
2023-01-11T22:59:41.0149137Z done
2023-01-11T22:59:41.0162877Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T22:59:41.0163156Z env:
2023-01-11T22:59:41.0163395Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:59:41.0163663Z   GPU_FLAG: --gpus all
2023-01-11T22:59:41.0164034Z   DOCKER_CONTAINER_ID: c5cb3f34fdb745cbab81ead9bbd8979c6cd118fca3d50c4cb582659c15cdaef3
2023-01-11T22:59:41.0164401Z ##[endgroup]
2023-01-11T22:59:41.0194033Z Holding runner for 2 hours until all ssh sessions have logged out
2023-01-11T22:59:41.0240992Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2023-01-11T22:59:41.0241399Z # ignore expansion of "docker ps -q" since it could be empty
2023-01-11T22:59:41.0241733Z # shellcheck disable=SC2046
2023-01-11T22:59:41.0242036Z docker stop $(docker ps -q) || true
2023-01-11T22:59:41.0242353Z # Prune all of the docker images
2023-01-11T22:59:41.0242627Z docker system prune -af
2023-01-11T22:59:41.0254625Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T22:59:41.0254919Z env:
2023-01-11T22:59:41.0255156Z   GIT_DEFAULT_BRANCH: master
2023-01-11T22:59:41.0255405Z   GPU_FLAG: --gpus all
2023-01-11T22:59:41.0255784Z   DOCKER_CONTAINER_ID: c5cb3f34fdb745cbab81ead9bbd8979c6cd118fca3d50c4cb582659c15cdaef3
2023-01-11T22:59:41.0256154Z ##[endgroup]
2023-01-11T22:59:41.4190777Z c5cb3f34fdb7
2023-01-11T22:59:42.5030155Z Deleted Containers:
2023-01-11T22:59:42.5030618Z c5cb3f34fdb745cbab81ead9bbd8979c6cd118fca3d50c4cb582659c15cdaef3
2023-01-11T22:59:42.5030884Z 
2023-01-11T22:59:47.2805392Z Deleted Images:
2023-01-11T22:59:47.2806273Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T22:59:47.2807254Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7@sha256:0da23f4faf0ce20770149c4a783e08eaa91c07112511dc5511c77937c66edb24
2023-01-11T22:59:47.2807894Z deleted: sha256:dd055998e88c3bb7db98caef99cc4aaaa492114a459a38a5f0ab49c735f40318
2023-01-11T22:59:47.2808351Z deleted: sha256:e4008aa27d9451086197883cfac22b827879bbe380f63c8c39e3db8313773f3c
2023-01-11T22:59:47.2808793Z deleted: sha256:acc638ed73c788f1c8fdbf04e65d27fa42e6c32d67dbdb50616e173ef284a563
2023-01-11T22:59:47.2809476Z deleted: sha256:f5db2d6ac11f27c63a5f2d0250a45efbd078c37a32d2e2973e544ea526501ba9
2023-01-11T22:59:47.2809892Z deleted: sha256:ce58ef265c69d549d5071cb5418ec43a703978b2c7a88b1141673272cd29de77
2023-01-11T22:59:47.2810312Z deleted: sha256:2e1d5c5ea4a63305e9617c6ab380960f330e1755c39c714f3a3eefb2b603e92a
2023-01-11T22:59:47.2810719Z deleted: sha256:f6e5b9727392978f694412774dde4231ab32666b8604ff0d64727308d45a9163
2023-01-11T22:59:47.2811144Z deleted: sha256:0cfd1beb2f29eb03cb53178c729bf68f37f74bbd713aa6e3a6b1dd0b8121eb61
2023-01-11T22:59:47.2811676Z deleted: sha256:8ec9997c9a1e58cb7d553f0862f1f975bdc1bbd0d0297b6c483bf8731508824b
2023-01-11T22:59:47.2812096Z deleted: sha256:367991b9b8c74307719e7037512b4d1d67917a888c40c4ed86af9843f0c38c77
2023-01-11T22:59:47.2812494Z deleted: sha256:83b736f4ed1139df1136ab38e29568f048e4ce111951d4fdc1a508713303ca00
2023-01-11T22:59:47.2812910Z deleted: sha256:d2ca56a8d719ce8cfe4b65b165cdee296b1b456c56eb3e06990ca62ba42bc18a
2023-01-11T22:59:47.2813356Z deleted: sha256:cdfc9a191a57609f700ced389b19973ef4436b66b1d47313c799be166d6fce4b
2023-01-11T22:59:47.2813776Z deleted: sha256:293a82b424c32cb77ae038879dff6856d5fd08e5b9db2c5b015f424ebf88d24b
2023-01-11T22:59:47.2814203Z deleted: sha256:21a63ac2015ba16ccb51b9090e0f4b78a8437ca4dd189f376d1e9f45e6c74d3e
2023-01-11T22:59:47.2814613Z deleted: sha256:afc58208e3d75e15f79b0f474b361bbc1e21b8eb232fe613a1c79b8415827c86
2023-01-11T22:59:47.2815026Z deleted: sha256:86179776c9e36c00cf4c4a64f717303555b00dc594ab664fb29cfdb707eece2b
2023-01-11T22:59:47.2815421Z deleted: sha256:45f06a2900391711a143b22a31156d3013a0f37430ff8367ab4dbb27ac33381e
2023-01-11T22:59:47.2815812Z deleted: sha256:d7d350150f2a9a9ac81a59d44fd01f74d44f5f354f07a2f055aea97c8be52d92
2023-01-11T22:59:47.2816312Z deleted: sha256:eae59ad1f09ea2b1f99a568bfd5f85dae44a3af9b46b5ac5af19931de9e8fb8d
2023-01-11T22:59:47.2816764Z deleted: sha256:d734a02f47eda33d1022c9642ccfc469b44f5b104ab8ae7aa855ea76be550288
2023-01-11T22:59:47.2817213Z deleted: sha256:643cbebfdfe0cee71fed16a3233b2ee4b6392d91833a41269cdc80c8d0841ae9
2023-01-11T22:59:47.2817647Z deleted: sha256:3a4db9e7d414af3a157be5297f5b7bcddc4c63c0e83221d36728ddc32f84bc2d
2023-01-11T22:59:47.2818094Z deleted: sha256:d93fc8ff6f3d356d012ef7cca3d02018fa3c0a4be9ca2ae4ce78d174274ce530
2023-01-11T22:59:47.2818546Z deleted: sha256:cdd9af1450d9ea8f070e7c03dfe076d49ab615a4d3558be68bd7a4f16d804038
2023-01-11T22:59:47.2818962Z deleted: sha256:de611f0f78a22b1c6c68620370fb99a959c668d8c38a2cc832912e784f94869f
2023-01-11T22:59:47.2819381Z deleted: sha256:0c7878e2d089271e8e5c181eef49d0a43c99b827d1a60dafd15a54627d9146b4
2023-01-11T22:59:47.2819803Z deleted: sha256:55fe9bdaa629b37baee75e1f6878ab1941e000e0937ef45452a9482f29de4577
2023-01-11T22:59:47.2820239Z deleted: sha256:1e641da8b29087f06ad852506d00c1eaaedaeed0bf2a451c1d462c0b476c169f
2023-01-11T22:59:47.2820645Z deleted: sha256:77079a0a22a485736fdd6052b5270d4fb8ee1771976ed0d55e0f315cbc6d1da5
2023-01-11T22:59:47.2821092Z deleted: sha256:b3dda998ed6389c88d249f3aa8d96b2f94876931c45284ef00425cdea77b7c07
2023-01-11T22:59:47.2821524Z deleted: sha256:7dcf69834443047244edadb0a7016bc391279d485de12e8d2d8aad25af532912
2023-01-11T22:59:47.2821954Z deleted: sha256:932db4f0d0b27a0b32d81f5e108fdec1c5433e7b4614cfdc39d64189a59bc228
2023-01-11T22:59:47.2822376Z deleted: sha256:99ea7bfa823ad0725ffcc42e5dd47b90270c3a3e49a0b2a31adae4497d029331
2023-01-11T22:59:47.2822816Z deleted: sha256:58c3ab544a412f163fd3613ab991ea85fa1ae7c97f5a6cbc7b86a1b97fdd5484
2023-01-11T22:59:47.2823250Z deleted: sha256:fd75951227e5b4de5b96f5ee360cb3f1caf2f32a33ca89976013a385d23a34be
2023-01-11T22:59:47.2823662Z deleted: sha256:980d2a371b758adf16fc78370ed6b8bcf77846721ef4f20da94a6d1299457ad6
2023-01-11T22:59:47.2824076Z deleted: sha256:311c72e743e14e8490d04f4331dcc0a35309e9b94266986b0b5badd5fe499765
2023-01-11T22:59:47.2824471Z deleted: sha256:857d2698f0a0af2341318f6ed93060f21d498e989b339e2867c5236bab9c63d5
2023-01-11T22:59:47.2824878Z deleted: sha256:df69730c501d9b3ce0f2316b2b638e20515ce1e9aad01098b13234f4a2154927
2023-01-11T22:59:47.2825351Z deleted: sha256:288a6d7efd5c9e470341b16b3ea2cd41124c769de8d10643ac688d651e9767c9
2023-01-11T22:59:47.2825766Z deleted: sha256:6a565f7b04668396c32d11cad845e1cd1b84d09e6f970457c5494adfde59690f
2023-01-11T22:59:47.2826195Z deleted: sha256:b4e45ce76a79fe9f3dbbf723401c4bf189f0e70bc81fc2dbe4e70c80044c2fac
2023-01-11T22:59:47.2826618Z deleted: sha256:e4a3cd7f0f84ce7171b04c994248707950295feb437bc5feaadcb66a3f7bf5a3
2023-01-11T22:59:47.2827040Z deleted: sha256:24ac908f9f592af03d6a51011147428f9e682c79b8ca2ad5afd2b1a44aeed617
2023-01-11T22:59:47.2827526Z deleted: sha256:50cc7186fda7f64aa964824a32c139a1085ce030491c0da5fab99fdecae66fdf
2023-01-11T22:59:47.2827938Z deleted: sha256:6e9c802974cfa887b7300715840ef7aaf5765df415d8d4680f72c6034a10292b
2023-01-11T22:59:47.2828336Z deleted: sha256:30109e8b967541225d66d116256e57334c2b63c25b456f9f7cd72d14d46d8da3
2023-01-11T22:59:47.2828990Z deleted: sha256:18ce8ec73f72efbc789a00688f5d57c798690e22048389f236dbc593cec31d6e
2023-01-11T22:59:47.2829693Z deleted: sha256:195741932e0b070b4fed22eee8d97719dc71f1f569594b418d777b87dbe76a6d
2023-01-11T22:59:47.2830096Z deleted: sha256:6f099faae794c47a468400004f89aed66ec84fa1bd6c606a9877ab09c84a5289
2023-01-11T22:59:47.2830516Z deleted: sha256:5bddaa98761511a0e16047132a49704d0cf176bec84f42b91644b8e7adb3cb88
2023-01-11T22:59:47.2830909Z deleted: sha256:5089072a88c6788d2594696a16346c495f97fd117430602f033541a0f333de5f
2023-01-11T22:59:47.2831284Z deleted: sha256:9bc67bb187c368480f186819831faa7998ba6d4f2e4ab8bd5b5fbc8a5aada045
2023-01-11T22:59:47.2831694Z deleted: sha256:45bbe3d22998589317c7f6c4dd591475423bb37ca9b922529c5878653483b18d
2023-01-11T22:59:47.2832582Z 
2023-01-11T22:59:47.2832806Z Total reclaimed space: 19.53GB
2023-01-11T22:59:47.2889250Z Post job cleanup.
2023-01-11T22:59:47.2926378Z Post job cleanup.
2023-01-11T22:59:47.4289581Z [command]/usr/bin/git version
2023-01-11T22:59:47.4385387Z git version 2.38.1
2023-01-11T22:59:47.4447805Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/d7d6812e-9082-4074-8597-d12bc81d867b' before making global git config changes
2023-01-11T22:59:47.4449286Z Adding repository directory to the temporary git global config as a safe directory
2023-01-11T22:59:47.4454445Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T22:59:47.4534344Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2023-01-11T22:59:47.4572365Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2023-01-11T22:59:47.4896810Z Entering 'android/libs/fbjni'
2023-01-11T22:59:47.4940803Z Entering 'third_party/FP16'
2023-01-11T22:59:47.4982763Z Entering 'third_party/FXdiv'
2023-01-11T22:59:47.5025119Z Entering 'third_party/NNPACK'
2023-01-11T22:59:47.5067496Z Entering 'third_party/QNNPACK'
2023-01-11T22:59:47.5111047Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T22:59:47.5153374Z Entering 'third_party/XNNPACK'
2023-01-11T22:59:47.5208581Z Entering 'third_party/benchmark'
2023-01-11T22:59:47.5250659Z Entering 'third_party/cpuinfo'
2023-01-11T22:59:47.5295462Z Entering 'third_party/cub'
2023-01-11T22:59:47.5339305Z Entering 'third_party/cudnn_frontend'
2023-01-11T22:59:47.5390530Z Entering 'third_party/cutlass'
2023-01-11T22:59:47.5441383Z Entering 'third_party/eigen'
2023-01-11T22:59:47.5493004Z Entering 'third_party/fbgemm'
2023-01-11T22:59:47.5538951Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T22:59:47.5582450Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T22:59:47.5627188Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T22:59:47.5670387Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T22:59:47.5713958Z Entering 'third_party/flatbuffers'
2023-01-11T22:59:47.5763232Z Entering 'third_party/fmt'
2023-01-11T22:59:47.5806146Z Entering 'third_party/foxi'
2023-01-11T22:59:47.5848808Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T22:59:47.5892016Z Entering 'third_party/gloo'
2023-01-11T22:59:47.5935885Z Entering 'third_party/googletest'
2023-01-11T22:59:47.5981356Z Entering 'third_party/ideep'
2023-01-11T22:59:47.6024907Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T22:59:47.6071063Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T22:59:47.6120458Z Entering 'third_party/ios-cmake'
2023-01-11T22:59:47.6163673Z Entering 'third_party/ittapi'
2023-01-11T22:59:47.6206731Z Entering 'third_party/kineto'
2023-01-11T22:59:47.6249007Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T22:59:47.6291754Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T22:59:47.6335912Z Entering 'third_party/nccl/nccl'
2023-01-11T22:59:47.6379519Z Entering 'third_party/neon2sse'
2023-01-11T22:59:47.6424461Z Entering 'third_party/nlohmann'
2023-01-11T22:59:47.6471019Z Entering 'third_party/onnx'
2023-01-11T22:59:47.6527887Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T22:59:47.6571920Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T22:59:47.6619135Z Entering 'third_party/onnx-tensorrt'
2023-01-11T22:59:47.6663818Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T22:59:47.6714314Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T22:59:47.6760733Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T22:59:47.6804197Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T22:59:47.6854456Z Entering 'third_party/pocketfft'
2023-01-11T22:59:47.6899548Z Entering 'third_party/protobuf'
2023-01-11T22:59:47.6948431Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T22:59:47.6992447Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T22:59:47.7039898Z Entering 'third_party/psimd'
2023-01-11T22:59:47.7084774Z Entering 'third_party/pthreadpool'
2023-01-11T22:59:47.7129573Z Entering 'third_party/pybind11'
2023-01-11T22:59:47.7174807Z Entering 'third_party/python-enum'
2023-01-11T22:59:47.7219172Z Entering 'third_party/python-peachpy'
2023-01-11T22:59:47.7261709Z Entering 'third_party/python-six'
2023-01-11T22:59:47.7304869Z Entering 'third_party/sleef'
2023-01-11T22:59:47.7348917Z Entering 'third_party/tbb'
2023-01-11T22:59:47.7395812Z Entering 'third_party/tensorpipe'
2023-01-11T22:59:47.7439782Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T22:59:47.7482416Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T22:59:47.7526317Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T22:59:47.7570396Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T22:59:47.7611292Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T22:59:47.7656500Z Entering 'third_party/zstd'
2023-01-11T22:59:47.7719322Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2023-01-11T22:59:47.7749208Z http.https://github.com/.extraheader
2023-01-11T22:59:47.7759611Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2023-01-11T22:59:47.7796497Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2023-01-11T22:59:47.8117896Z Entering 'android/libs/fbjni'
2023-01-11T22:59:47.8143833Z http.https://github.com/.extraheader
2023-01-11T22:59:47.8179652Z Entering 'third_party/FP16'
2023-01-11T22:59:47.8205666Z http.https://github.com/.extraheader
2023-01-11T22:59:47.8241435Z Entering 'third_party/FXdiv'
2023-01-11T22:59:47.8267183Z http.https://github.com/.extraheader
2023-01-11T22:59:47.8301816Z Entering 'third_party/NNPACK'
2023-01-11T22:59:47.8328467Z http.https://github.com/.extraheader
2023-01-11T22:59:47.8363441Z Entering 'third_party/QNNPACK'
2023-01-11T22:59:47.8389606Z http.https://github.com/.extraheader
2023-01-11T22:59:47.8425366Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T22:59:47.8452335Z http.https://github.com/.extraheader
2023-01-11T22:59:47.8488220Z Entering 'third_party/XNNPACK'
2023-01-11T22:59:47.8515443Z http.https://github.com/.extraheader
2023-01-11T22:59:47.8563028Z Entering 'third_party/benchmark'
2023-01-11T22:59:47.8588764Z http.https://github.com/.extraheader
2023-01-11T22:59:47.8623761Z Entering 'third_party/cpuinfo'
2023-01-11T22:59:47.8650638Z http.https://github.com/.extraheader
2023-01-11T22:59:47.8685323Z Entering 'third_party/cub'
2023-01-11T22:59:47.8711154Z http.https://github.com/.extraheader
2023-01-11T22:59:47.8744764Z Entering 'third_party/cudnn_frontend'
2023-01-11T22:59:47.8771441Z http.https://github.com/.extraheader
2023-01-11T22:59:47.8812029Z Entering 'third_party/cutlass'
2023-01-11T22:59:47.8845415Z http.https://github.com/.extraheader
2023-01-11T22:59:47.8888969Z Entering 'third_party/eigen'
2023-01-11T22:59:47.8914654Z http.https://github.com/.extraheader
2023-01-11T22:59:47.8949545Z Entering 'third_party/fbgemm'
2023-01-11T22:59:47.8976084Z http.https://github.com/.extraheader
2023-01-11T22:59:47.9009741Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T22:59:47.9034570Z http.https://github.com/.extraheader
2023-01-11T22:59:47.9067292Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T22:59:47.9094380Z http.https://github.com/.extraheader
2023-01-11T22:59:47.9128273Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T22:59:47.9152399Z http.https://github.com/.extraheader
2023-01-11T22:59:47.9185690Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T22:59:47.9210761Z http.https://github.com/.extraheader
2023-01-11T22:59:47.9245162Z Entering 'third_party/flatbuffers'
2023-01-11T22:59:47.9270001Z http.https://github.com/.extraheader
2023-01-11T22:59:47.9306043Z Entering 'third_party/fmt'
2023-01-11T22:59:47.9331880Z http.https://github.com/.extraheader
2023-01-11T22:59:47.9365733Z Entering 'third_party/foxi'
2023-01-11T22:59:47.9390618Z http.https://github.com/.extraheader
2023-01-11T22:59:47.9424656Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T22:59:47.9450519Z http.https://github.com/.extraheader
2023-01-11T22:59:47.9484046Z Entering 'third_party/gloo'
2023-01-11T22:59:47.9509538Z http.https://github.com/.extraheader
2023-01-11T22:59:47.9543366Z Entering 'third_party/googletest'
2023-01-11T22:59:47.9569399Z http.https://github.com/.extraheader
2023-01-11T22:59:47.9602733Z Entering 'third_party/ideep'
2023-01-11T22:59:47.9627654Z http.https://github.com/.extraheader
2023-01-11T22:59:47.9660637Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T22:59:47.9687298Z http.https://github.com/.extraheader
2023-01-11T22:59:47.9722576Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T22:59:47.9747316Z http.https://github.com/.extraheader
2023-01-11T22:59:47.9788175Z Entering 'third_party/ios-cmake'
2023-01-11T22:59:47.9814266Z http.https://github.com/.extraheader
2023-01-11T22:59:47.9847804Z Entering 'third_party/ittapi'
2023-01-11T22:59:47.9873047Z http.https://github.com/.extraheader
2023-01-11T22:59:47.9907089Z Entering 'third_party/kineto'
2023-01-11T22:59:47.9934418Z http.https://github.com/.extraheader
2023-01-11T22:59:47.9967540Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T22:59:47.9994670Z http.https://github.com/.extraheader
2023-01-11T22:59:48.0036203Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T22:59:48.0061030Z http.https://github.com/.extraheader
2023-01-11T22:59:48.0095796Z Entering 'third_party/nccl/nccl'
2023-01-11T22:59:48.0121872Z http.https://github.com/.extraheader
2023-01-11T22:59:48.0155303Z Entering 'third_party/neon2sse'
2023-01-11T22:59:48.0179597Z http.https://github.com/.extraheader
2023-01-11T22:59:48.0212313Z Entering 'third_party/nlohmann'
2023-01-11T22:59:48.0237547Z http.https://github.com/.extraheader
2023-01-11T22:59:48.0271346Z Entering 'third_party/onnx'
2023-01-11T22:59:48.0296939Z http.https://github.com/.extraheader
2023-01-11T22:59:48.0344278Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T22:59:48.0370180Z http.https://github.com/.extraheader
2023-01-11T22:59:48.0404689Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T22:59:48.0428991Z http.https://github.com/.extraheader
2023-01-11T22:59:48.0466025Z Entering 'third_party/onnx-tensorrt'
2023-01-11T22:59:48.0492013Z http.https://github.com/.extraheader
2023-01-11T22:59:48.0525932Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T22:59:48.0550511Z http.https://github.com/.extraheader
2023-01-11T22:59:48.0589712Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T22:59:48.0621046Z http.https://github.com/.extraheader
2023-01-11T22:59:48.0655415Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T22:59:48.0680817Z http.https://github.com/.extraheader
2023-01-11T22:59:48.0714575Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T22:59:48.0739535Z http.https://github.com/.extraheader
2023-01-11T22:59:48.0778729Z Entering 'third_party/pocketfft'
2023-01-11T22:59:48.0804982Z http.https://github.com/.extraheader
2023-01-11T22:59:48.0838442Z Entering 'third_party/protobuf'
2023-01-11T22:59:48.0863194Z http.https://github.com/.extraheader
2023-01-11T22:59:48.0900420Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T22:59:48.0925540Z http.https://github.com/.extraheader
2023-01-11T22:59:48.0958898Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T22:59:48.0983165Z http.https://github.com/.extraheader
2023-01-11T22:59:48.1017792Z Entering 'third_party/psimd'
2023-01-11T22:59:48.1043452Z http.https://github.com/.extraheader
2023-01-11T22:59:48.1077220Z Entering 'third_party/pthreadpool'
2023-01-11T22:59:48.1101544Z http.https://github.com/.extraheader
2023-01-11T22:59:48.1135589Z Entering 'third_party/pybind11'
2023-01-11T22:59:48.1162614Z http.https://github.com/.extraheader
2023-01-11T22:59:48.1195220Z Entering 'third_party/python-enum'
2023-01-11T22:59:48.1219419Z http.https://github.com/.extraheader
2023-01-11T22:59:48.1251985Z Entering 'third_party/python-peachpy'
2023-01-11T22:59:48.1277350Z http.https://github.com/.extraheader
2023-01-11T22:59:48.1309038Z Entering 'third_party/python-six'
2023-01-11T22:59:48.1334283Z http.https://github.com/.extraheader
2023-01-11T22:59:48.1366151Z Entering 'third_party/sleef'
2023-01-11T22:59:48.1391070Z http.https://github.com/.extraheader
2023-01-11T22:59:48.1424144Z Entering 'third_party/tbb'
2023-01-11T22:59:48.1449396Z http.https://github.com/.extraheader
2023-01-11T22:59:48.1484923Z Entering 'third_party/tensorpipe'
2023-01-11T22:59:48.1509379Z http.https://github.com/.extraheader
2023-01-11T22:59:48.1542399Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T22:59:48.1567165Z http.https://github.com/.extraheader
2023-01-11T22:59:48.1599851Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T22:59:48.1624073Z http.https://github.com/.extraheader
2023-01-11T22:59:48.1657447Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T22:59:48.1683010Z http.https://github.com/.extraheader
2023-01-11T22:59:48.1716552Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T22:59:48.1740465Z http.https://github.com/.extraheader
2023-01-11T22:59:48.1773148Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T22:59:48.1798265Z http.https://github.com/.extraheader
2023-01-11T22:59:48.1841610Z Entering 'third_party/zstd'
2023-01-11T22:59:48.1866475Z http.https://github.com/.extraheader
2023-01-11T22:59:48.2178941Z Cleaning up orphan processes